Neural network applications in resource constrained environments

ABSTRACT

Systems and methods are disclosed for applying neural networks in resource-constrained environments. A system may include a sensor located in a resource-constrained environment configured to generate first sensor data and second sensor data of the resource-constrained environment. The system may also include a first computing device not located in the resource-constrained environment configured to produce a neural network structure based on the first sensor data. The system may also include a second computing device configured to determine a state of the resource-constrained environment based on input of the second sensor data to the neural network structure. The system may also include a controller located in the resource-constrained environment configured to control a device in the resource-constrained environment based on the state of the resource-constrained environment determined by the second computing device. The second computing device may be further configured to calculate an activation area for the neural network structure.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.15/699,298, filed on Sep. 8, 2017, entitled “NEURAL NETWORK APPLICATIONSIN RESOURCE CONSTRAINED ENVIRONMENTS,” which is a continuation of U.S.application Ser. No. 15/397,715, filed on Jan. 3, 2017, entitled “NEURALNETWORK APPLICATIONS IN RESOURCE CONSTRAINED ENVIRONMENTS,” now U.S.Pat. No. 9,760,827, which claims priority to U.S. ProvisionalApplication 62/365,629 filed on Jul. 22, 2016 entitled “NEURAL NETWORKSFOR USE IN AUTOMOTIVE SYSTEMS”, which are incorporated herein in theirentirety.

TECHNICAL FIELD

This patent document relates to systems, devices, and processes that useneural networks in resource-constrained environments.

BACKGROUND

Artificial neural networks are processing devices that are somewhatmodeled after the neural structure of a cerebral cortex. Neural networksperform a type of supervised learning, using known inputs and knownoutputs to develop a model to categorize, classify, or predict somethingabout a future input. Neural networks are highly adaptable and thus havebeen employed in many different domains.

SUMMARY

According to some embodiments of the present disclosure, a system isprovided. The system includes a sensor located in a resource-constrainedenvironment configured to generate sensor data of theresource-constrained environment. The system further includes a firstcomputing device not located in the resource-constrained environmentconfigured to produce a neural network structure based on the sensordata. The system further includes a second computing device located inthe resource-constrained environment configured to provide the sensordata as input to the neural network structure. The second computingdevice is further configured to determine a state of theresource-constrained environment based on the input of the sensor datato the neural network structure.

In some embodiments, the system further includes a controller located inthe resource-constrained environment configured to control an element inthe resource-constrained environment based on the state of theresource-constrained environment determined by the second computingdevice.

In some embodiments, the system further includes a transceiver locatedin the resource-constrained environment configured to transmit thesensor data from the sensor to the first computing device.

In some embodiments, the transceiver is configured to transmit thesensor data from the sensor to the first computing device using awireless communications link.

In some embodiments, the wireless communications link providesintermittent communications between the transceiver and the firstcomputing device.

In some embodiments, the sensor is located in or next to a rearviewmirror of an automobile.

In some embodiments, the sensor includes a camera and the sensor dataincludes visual images of the resource-constrained environment.

In some embodiments, the first computing device is configured to producethe neural network structure based in part on labels associated with thesensor data.

In some embodiments, the labels associated with the sensor data identifyone or more states of the resource-constrained environment as capturedby the sensor data.

In some embodiments, the system further includes an automobilecontroller located in the resource-constrained environment configured tocontrol a self-driving mode of an automobile based on the state of theresource-constrained environment determined by the second computingdevice.

In some embodiments, the second computing device is configured todetermine whether a driver in the resource-constrained environment is ina safe driving state based on the input of the sensor data to the neuralnetwork structure.

In some embodiments, the system further includes an automobilecontroller located in the resource-constrained environment configured tocontrol an indicator device of an automobile based on the state of theresource-constrained environment determined by the second computingdevice.

In some embodiments, the second computing device is configured todetermine whether an adult is present in the resource-constrainedenvironment based on the input of the sensor data to the neural networkstructure. In such embodiments, the second computing device is furtherconfigured to determine whether a child is present in theresource-constrained environment based on the input of the sensor datato the neural network structure.

In some embodiments, the second computing device is configured todetermine whether a child is restrained in a child safety seat in theresource-constrained environment based on the input of the sensor datato the neural network structure.

In some embodiments, the second computing device is configured todetermine whether an item has been abandoned in the resource-constrainedenvironment based on the input of the sensor data to the neural networkstructure.

According to some embodiments of the present disclosure, a method isprovided. The method includes generating sensor data of aresource-constrained environment using a sensor located in theresource-constrained environment. The method further includes producinga neural network structure based on the sensor data using a firstcomputing device not located in the resource-constrained environment.The method further includes inputting the sensor data to the neuralnetwork structure using a second computing device located in theresource-constrained environment. The method further includesdetermining a state of the resource-constrained environment based on aresult of the inputting of the sensor data to the neural networkstructure.

According to some embodiments of the present disclosure, a system isprovided. The system includes a sensor located in an automobile. In suchembodiments, the sensor is configured to produce first sensor datarelated to an image of an interior of the automobile and second sensordata related to an image of the interior of the automobile. The systemfurther includes a local transceiver device located in the automobile.In such embodiments, the local transceiver device is configured toreceive the first sensor data from the sensor. The system furtherincludes a remote computing device located remote from the automobile.In such embodiments, the remote computing device is configured to applya neural network to the first sensor data in order to produce neuralnetwork configuration parameters. The system further includes acommunication link provided between the local transceiver device and theremote computing device. In such embodiments, the local transceiverdevice is configured to transmit the first sensor data over thecommunication link to the remote computing device. In such embodiments,the local transceiver device is configured to receive the neural networkconfiguration parameters over the communication link from the remotecomputing device. The system further includes a local computing devicelocated in the automobile. In such embodiments, the local computingdevice is configured to receive the neural network configurationparameters from the local transceiver device. In such embodiments, thelocal computing device is configured to receive the second sensor datafrom the sensor. In such embodiments, the local computing device isconfigured to determine whether a driver of the automobile is in adistracted state based on the neural network configuration parametersand the second sensor data. The system further includes an automobilecontroller located in the automobile. In such embodiments, theautomobile controller is configured to control a transition of theautomobile between a self-driving mode and a non-self-driving mode. Insuch embodiments, the automobile controller is configured to receive anoutput of the distracted state determination from the local computingdevice. In such embodiments, the automobile controller is configured torestrict the transition of the automobile between the self-driving modeand the non-self-driving mode based on the output of the distractedstate determination.

According to some embodiments of the present disclosure, a system isprovided. The system includes a sensor located in an automobile. In suchembodiments, the sensor is configured to produce first sensor datarelated to an image of an interior of the automobile and second sensordata related to an image of the interior of the automobile. The systemfurther includes a local transceiver device located in the automobile.In such embodiments, the local transceiver device is configured toreceive the first sensor data from the sensor. The system furtherincludes a remote computing device located remote from the automobile.In such embodiments, the remote computing device is configured to applya neural network to the first sensor data in order to produce neuralnetwork configuration parameters. The system further includes acommunication link provided between the local transceiver device and theremote computing device. In such embodiments, the local transceiverdevice is configured to transmit the first sensor data over thecommunication link to the remote computing device. In such embodiments,the local transceiver device is configured to receive the neural networkconfiguration parameters over the communication link from the remotecomputing device. The system further includes a local computing devicelocated in the automobile. In such embodiments, the local computingdevice is configured to receive the neural network configurationparameters from the local transceiver device. In such embodiments, thelocal computing device is configured to receive the second sensor datafrom the sensor. In such embodiments, the local computing device isconfigured to determine whether a child is present in a child seat andwhether an adult is present in a car seat based on the neural networkconfiguration parameters and the second sensor data. The system furtherincludes an automobile controller located in the automobile. In suchembodiments, the automobile controller is configured to control anindicator device. In such embodiments, the automobile controller isconfigured to receive an output of the determination of whether a childis present in the child seat and whether an adult is present in the carseat. In such embodiments, the automobile controller is configured tocause the indicator device to output an alert based on the determinationof whether a child is present in the child seat and whether an adult ispresent in the car seat.

According to some embodiments of the present disclosure, a system isprovided. The system includes a first sensor located in an automobile.In such embodiments, the first sensor is configured to produce firstsensor data related to an image of an interior of the automobile andsecond sensor data related to an image of the interior of theautomobile. The system further includes a second sensor located in anautomobile. In such embodiments, the second sensor is configured toproduce third sensor data related to an image of an interior of theautomobile and fourth sensor data related to an image of the interior ofthe automobile. The system further includes a local transceiver devicelocated in the automobile. In such embodiments, the local transceiverdevice is configured to receive the first sensor data from the firstsensor. In such embodiments, the local transceiver device is configuredto receive the third sensor data from the second sensor. The systemfurther includes a remote computing device located remote from theautomobile. In such embodiments, the remote computing device isconfigured to apply a neural network to the first sensor data in orderto produce first neural network configuration parameters. In suchembodiments, the remote computing device is configured to apply a neuralnetwork to the third sensor data in order to produce second neuralnetwork configuration parameters. The system further includes acommunication link provided between the local transceiver device and theremote computing device. In such embodiments, the local transceiverdevice is configured to transmit the first sensor data over thecommunication link to the remote computing device. In such embodiments,the local transceiver device is configured to transmit the third sensordata over the communication link to the remote computing device. In suchembodiments, the local transceiver device is configured to receive thefirst neural network configuration parameters over the communicationlink from the remote computing device. In such embodiments, the localtransceiver device is configured to receive the second neural networkconfiguration parameters over the communication link from the remotecomputing device. The system further includes a local computing devicelocated in the automobile. In such embodiments, the local computingdevice is configured to receive the first neural network configurationparameters from the local transceiver device. In such embodiments, thelocal computing device is configured to receive the second neuralnetwork configuration parameters from the local transceiver device. Insuch embodiments, the local computing device is configured to receivethe second sensor data from the first sensor. In such embodiments, thelocal computing device is configured to receive the fourth sensor datafrom the second sensor. In such embodiments, the local computing deviceis configured to determine whether an adult of the automobile is in apresent state or an absent state based on the first neural networkconfiguration parameters and the second sensor data. In suchembodiments, the local computing device is configured to determinewhether a child is in a present state or an absent state based on thesecond neural network configuration parameters and the fourth sensordata. The system further includes an automobile controller located inthe automobile. In such embodiments, the automobile controller isconfigured to control an indicator device. In such embodiments, theautomobile controller is configured to receive an output of the adultpresent or absent determination from the local computing device. In suchembodiments, the automobile. In such embodiments, the automobilecontroller is configured to receive an output of the child present orabsent determination from the local computing device. In suchembodiments, the automobile controller is configured to cause theindicator device to output an alert based on the output of the childpresent or absent determination indicating that the child is in apresent state and based on the output of the adult present or absentdetermination indicating that the adult was in a present statepreviously and is now in a present state.

According to some embodiments of the present disclosure, a system isprovided. The system includes a first sensor located in an automobile.In such embodiments, the first sensor is configured to produce firstsensor data related to an image of an interior of the automobile andsecond sensor data related to an image of the interior of theautomobile. The system further includes, a second sensor located in theautomobile. In such embodiments, the second sensor is configured toproduced third sensor data indicating a gear of a transmission of theautomobile. The system further includes a local transceiver devicelocated in the automobile. In such embodiments, the local transceiverdevice is configured to receive the first sensor data from the firstsensor. The system further includes a remote computing device locatedremote from the automobile. In such embodiments, the remote computingdevice is configured to apply a neural network to the first sensor datain order to produce neural network configuration parameters. The systemfurther includes a communication link provided between the localtransceiver device and the remote computing device. In such embodiments,the local transceiver device is configured to transmit the first sensordata over the communication link to the remote computing device. In suchembodiments, the local transceiver device is configured to receive theneural network configuration parameters over the communication link fromthe remote computing device. The system further includes a localcomputing device located in the automobile. In such embodiments, thelocal computing device is configured to receive the neural networkconfiguration parameters from the local transceiver device. In suchembodiments, the local computing device is configured to receive thesecond sensor data from the first sensor. In such embodiments, the localcomputing device is configured to determine whether a child is in arestrained state or a not restrained state based on the neural networkconfiguration parameters and the second sensor data. The system furtherincludes an automobile controller located in the automobile. In suchembodiments, the automobile controller is configured to control anindicator device. In such embodiments, the automobile controller isconfigured to receive an output of the determination of whether thechild is in a restrained state or a not restrained state. In suchembodiments, the automobile controller is configured to receive thethird sensor data from the second sensor. In such embodiments, theautomobile controller is configured to cause the indicator device tooutput an alert based on the output of the determination of whether thechild is in a restrained state or a not restrained state indicating thatthe child is in a not restrained state and based on the third sensordata indicating that the automobile is in a gear that is not the parkgear.

According to some embodiments of the present disclosure, a system isprovided. The system includes a first sensor located in an automobile.In such embodiments, the first sensor is configured to produce firstsensor data related to an image of an interior of the automobile andsecond sensor data related to an image of the interior of theautomobile. The system further includes a second sensor located in theautomobile. In such embodiments, the second sensor is configured toproduced third sensor data indicating a velocity of the automobile. Thesystem further includes a local transceiver device located in theautomobile. In such embodiments, the local transceiver device isconfigured to receive the first sensor data from the first sensor. Thesystem further includes a remote computing device located remote fromthe automobile. In such embodiments, the remote computing device isconfigured to apply a neural network to the first sensor data in orderto produce neural network configuration parameters. The system furtherincludes a communication link provided between the local transceiverdevice and the remote computing device. In such embodiments, the localtransceiver device is configured to transmit the first sensor data overthe communication link to the remote computing device. In suchembodiments, the local transceiver device is configured to receive theneural network configuration parameters over the communication link fromthe remote computing device. The system further includes a localcomputing device located in the automobile. In such embodiments, thelocal computing device is configured to receive the neural networkconfiguration parameters from the local transceiver device. In suchembodiments, the local computing device is configured to receive thesecond sensor data from the first sensor. In such embodiments, the localcomputing device is configured to determine whether a child is in arestrained state or a not restrained state based on the neural networkconfiguration parameters and the second sensor data. The system furtherincludes an automobile controller located in the automobile. In suchembodiments, the automobile controller is configured to control anindicator device. In such embodiments, the automobile controller isconfigured to receive an output of the determination of whether thechild is in a restrained state or a not restrained state. In suchembodiments, the automobile controller is configured to receive thethird sensor data from the second sensor. In such embodiments, theautomobile controller is configured to cause the indicator device tooutput an alert based on the output of the determination of whether thechild is in a restrained state or a not restrained state indicating thatthe child is in a not restrained state and based on the third sensordata indicating that the automobile has a velocity that is greater thanzero.

According to some embodiments of the present disclosure, a system isprovided. The system includes a sensor located in an automobile. In suchembodiments, the sensor is configured to produce first sensor datarelated to an image of an interior of the automobile and second sensordata related to an image of the interior of the automobile. The systemfurther includes a local transceiver device located in the automobile.In such embodiments, the local transceiver device is configured toreceive the first sensor data from the sensor. The system furtherincludes a remote computing device located remote from the automobile.In such embodiments, the remote computing device is configured to applya neural network to the first sensor data in order to produce neuralnetwork configuration parameters. The system further includes acommunication link provided between the local transceiver device and theremote computing device. In such embodiments, the local transceiverdevice is configured to transmit the first sensor data over thecommunication link to the remote computing device. In such embodiments,the local transceiver device is configured to receive the neural networkconfiguration parameters over the communication link from the remotecomputing device. The system further includes a local computing devicelocated in the automobile. In such embodiments, the local computingdevice is configured to receive the neural network configurationparameters from the local transceiver device. In such embodiments, thelocal computing device is configured to receive the second sensor datafrom the sensor. In such embodiments, the local computing device isconfigured to determine whether an abandoned item is present in theautomobile based on the neural network configuration parameters and thesecond sensor data. The system further includes an automobile controllerlocated in the automobile. In such embodiments, the automobilecontroller is configured to control an indicator device. In suchembodiments, the automobile controller is configured to receive anoutput of the determination of whether an abandoned item is present inthe automobile. In such embodiments, the automobile controller isconfigured to cause the indicator device to output an alert based on theoutput of the determination of whether an abandoned item is present inthe automobile.

In some embodiments, the sensor generates the first sensor data when theautomobile is in an unoccupied state. In such embodiments, the sensorgenerates the second sensor data after the automobile transitions froman occupied state to an unoccupied state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a resource-constrained environment in which aneural network may be employed.

FIG. 2 is a diagram of the interior of an automobile in which a neuralnetwork may be employed according to some embodiments of the presentdisclosure.

FIG. 3 is a schematic diagram from an overhead view of an automobile inwhich a neural network may be employed according to some embodiments ofthe present disclosure.

FIG. 4A is a schematic diagram from an overhead view of an automobile inwhich a neural network may be employed according to some embodiments ofthe present disclosure.

FIGS. 4B and 4C are illustrations of an interior of an automobileshowing the location of a sensor according to some embodiments of thepresent disclosure.

FIGS. 5A, 5B, 5C, 5D, and 5E are example images of a driver of anautomobile as captured by a sensor in an automobile according to someembodiments of the present disclosure.

FIGS. 6A, 6B, and 6C are diagrams of exemplary neural network structuresaccording to some embodiments of the present disclosure.

FIG. 7A is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 7B is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 8 is a diagram of a sensor device according to some embodiments ofthe present disclosure.

FIG. 9 is a diagram of a local transceiver according to some embodimentsof the present disclosure.

FIG. 10 is a diagram of a remote computing device according to someembodiments of the present disclosure.

FIG. 11 is a diagram of a local computing device according to someembodiments of the present disclosure.

FIG. 12 is a diagram of an automobile controller according to someembodiments of the present disclosure.

FIG. 13 is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 14 is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 15 is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 16 is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 17 is a system diagram of a system employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 18 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 19 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 20 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 21 is a flowchart of a process for employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 22 is a flowchart of a process for employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 23 is a flowchart of a process for employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 24 is a flowchart of a process for employing a neural network in anautomotive environment according to some embodiments of the presentdisclosure.

FIG. 25 is a diagram of an exemplary neural network structure accordingto some embodiments of the present disclosure.

FIGS. 26A and 26B are exemplary audio file formatting for input to aneural network structure according to some embodiments of the presentdisclosure.

FIG. 27 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 28 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 29 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 30 is a diagram of sensor data and difference calculationsaccording to some embodiments of the present disclosure.

FIG. 31 is a diagram of sensor data and difference calculationsaccording to some embodiments of the present disclosure.

FIG. 32 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

FIG. 33 is a diagram of sensor data and activation area calculationaccording to some embodiments of the present disclosure.

FIG. 34 is a diagram of sensor data and activation area calculationaccording to some embodiments of the present disclosure.

FIG. 35 is a diagram of sensor data and activation area expectationaccording to some embodiments of the present disclosure.

FIG. 36A is a flowchart of a process for segmenting sensor data for usewith a neural network in a resource-constrained environment according tosome embodiments of the present disclosure.

FIG. 36B is a flowchart of a process for segmenting sensor data for usewith a neural network in a resource-constrained environment according tosome embodiments of the present disclosure.

FIG. 37 is a flowchart of a process for segmenting sensor data for usewith a neural network in a resource-constrained environment according tosome embodiments of the present disclosure.

FIG. 38 is a diagram of sensor data and sensor data segmenting accordingto some embodiments of the present disclosure.

FIG. 39A is a flowchart of a process for training a neural network in aresource-constrained environment using keypoints according to someembodiments of the present disclosure.

FIG. 39B is a flowchart of a process for training a neural network in aresource-constrained environment using keyzones according to someembodiments of the present disclosure.

FIG. 40 is a diagram of an annotation interface according to someembodiments of the present disclosure.

FIG. 41 is a diagram of an annotation interface according to someembodiments of the present disclosure.

DETAILED DESCRIPTION

While neural networks are versatile and have been employed in manydomains, there are significant challenges to deploying them in certainenvironments. This is because neural networks traditionally requiresignificant processing power to train (i.e., to initially setup) andeven to use.

As with other types of machine learning, neural networks leverage largevolumes of input data in order to learn patterns about an environment.Also, as with other forms of machine learning, this process of learningpatterns from large volumes of data (“training”) can require bothsignificant data storage resources and significant computer processingresources. While efficient algorithms have been developed to performthis training of the neural network, the training nonetheless requiresmany steps of data processing. As such, significant processing power isoften used to speed up this training, such as with multi-coreprocessors, multiple computers operating concurrently, and other formsof high-powered computing.

The output of the training process for a neural network is a set ofconfiguration parameters that define the neural network and that reflecta model of the environment which the input data describes. The neuralnetwork defined by these configuration parameters can then be used withnew input data in order to classify, categorize, or predict some valuefor the new input data.

Contrary to some other types of machine learning, neural networks canalso require significant computing resources to use on new input data,even after having been trained. With many types of machine learning, thetraining of the model requires significant computing resources. But, theuse of that model on new input data can be done with very littleprocessing power. To the contrary, a neural network can requiresignificant computing resources to use even after the configurationparameters are determined during the training process.

Therefore, an impediment exists to using neural networks in environmentswhere significant processing power is not available, that is, inresource-constrained environments. Resource-constrained environments mayinclude environments where embedded devices are used (e.g., embeddedprocessing devices in an automobile), in mobile environments (e.g., asmartphone), in an Internet of Things environment (e.g., a networkedcontroller for a refrigerator), and in a ubiquitous computingenvironment (e.g., a networked headset like GOOGLE GLASS). Because thedevices in these environments may only have minimal processing power,these devices have typically not been able to make use of neuralnetworks.

However, the present inventors recognized that these sameresource-constrained environments may be greatly improved by the use ofneural networks. For instance, the aforementioned examples of resourceconstrained environments involve high levels of human involvement, muchmore so than a typical high-powered computing environment. Due to thestrength of neural networks to learn patterns in human environments, thepresent inventors recognized that neural networks could effectsignificant improvements to the computing devices and systems deployedin these resource-constrained environments.

Resource-constrained environments have also been consideredinappropriate environments for application of neural networks due to theintermittent communications often associated with such environments.Because a computing device in a resource-constrained environment mayoperate on low power, it may not be feasible to have an always-availablecommunications link between the resource-constrained computing deviceand other computing devices. Further, because a computing device in aresource-constrained environment may be a low cost embedded device, itmay not be desirable to incur the financial cost and technical overheadof establishing an always-available communications link between thecomputing device and other computing devices. Further, because acomputing device in a resource-constrained environment may move aroundwidely, it may enter areas with reduced telecommunicationsinfrastructure (e.g., lack of Wi-Fi and/or cellular networks) or areaswith no authorized telecommunications infrastructure (e.g., outside therange of recognized Wi-Fi networks). This intermittent communicationsavailability common in many resource-constrained environments has beenconsidered an impediment to deploying neural networks, at least becauseit obstructed the ability to receive training data from the environmentand then provide a trained neural network structure to the environment.

However, the present inventors recognized that this obstacle ofintermittent communications in resource-constrained environments couldbe overcome with a variety of techniques. The present inventorsrecognized ways to not only deploy, but also to update, neural networksdeployed in resource-constrained environments.

For the purpose of clarity, several embodiments demonstrating thesebenefits of using neural networks in resource-constrained environmentsare now described. For these exemplary embodiments, an automotiveenvironment is used.

FIG. 1 is a diagram of a resource-constrained environment 100 in which aneural network may be employed. The environment 100 includes anautomobile 110, a pedestrian 112, and a bicyclist 114. In theenvironment 100, a driver may be operating the automobile 110. Thedriver may operate the automobile 110 so as to avoid safety hazards,such as a collision with the pedestrian 112 or a collision with thebicyclist 114. The driver may also manage various tasks within theautomobile 110, such as controlling music playback over the audio systemof the automobile 110 or assuring that other passengers of theautomobile 110 are securely fastened into a seat. The automobile 110 mayhave a self-driving or autopilot mode that allows the automobile 110and/or a controller thereof to perform some of the operating tasksgenerally performed by the driver. In some situations, this self-drivingor autopilot mode may provide a degree of autonomous driving by theautomobile 110. The automobile 110 may have various indicator systems(e.g., audible alarms, visual indicators in the instrument cluster) inorder to alert the driver to conditions or hazards in the environment100 (e.g., presence of pedestrian 112, failure of the driver to fastenhis seat belt).

The driver of the automobile 110 may possess a mobile device 120. Themobile device 120 may be present in the automobile 100 while the driveris operating the automobile 110. The mobile device 120 may communicateover a communication link 140 with remote networks and computing devicesdenoted by cloud 130. In some embodiments, a neural network may betrained in the cloud 130 and used in the automobile 110 in order toassist the driver in operating the automobile 110 and/or to assist theautomobile 110 in operating in a self-driving or autopilot mode.

FIG. 2 is a diagram of the interior 200 of an automobile in which aneural network may be employed according to some embodiments of thepresent disclosure. The interior 200 may include a driver's seat 211, apassenger's seat 212, and a steering wheel 213. The interior 200 mayfurther include an infotainment display 214 and an instrument cluster215. The driver of the automobile in which interior 200 is situated maysit in the driver's seat 211 and use the steering wheel 213,infotainment display 214, instrument cluster 215, and other elements inthe interior 200 to operate the automobile.

The interior 200 may further include sensor 232 and sensor 234. Sensor232 may be provided built into the steering wheel 213. Sensor 234 may beprovided built into a center console of the interior 200. The sensors232 and 234 may be configured to sense the interior 200. For instance,the sensors may sense the presence or absence of a driver in thedriver's seat 211, the presence or absence of a passenger in thepassenger's seat 212, the position of the arms and hands of a driverseated in the driver's seat 211, the orientation of the head of a driverseated in the driver's seat 211, and the presence of an object presentin the interior 200 (e.g., soda can 242). The sensors 232 and 234 may beprovided in a variety of forms, such as a video camera, an infraredemitter and sensor, and/or an ultrasound emitter and sensor.

FIG. 3 is an schematic diagram from an overhead view of an automobile300 in which a neural network may be employed according to someembodiments of the present disclosure. The automobile 300 may include adriver's seat 301, a passenger's seat 302, a rear seat 303, a childsafety seat 304, a steering wheel 305, an instrument cluster 306, aninfotainment display 307, a dashboard 308, and a center console 309.These features may be provided as described previously and as used inordinary automobile environments.

The automobile 300 may further include various locations at whichsensors may be located. A sensor may be located at location 351 builtinto the steering wheel 305. A sensor may be located at location 352built into the instrument cluster 306. A sensor may be located atlocation 353 provided on or under a rearview mirror. A sensor may belocated at location 354 built into the dashboard 308. A sensor may beprovided at location 355 built into the infotainment display 307. Asensor may be provided at location 356 built into a dome light unit onthe interior ceiling of the automobile 300. A sensor may be provided atlocation 357 built into the center console 309. A sensor may be providedat location 358 built into a dome light unit on the interior ceiling theautomobile 300. A sensor may be provided at location 359 affixed to theinterior ceiling the automobile 300. A sensor may be provided atlocation 360 affixed to the interior ceiling the automobile 300. Asensor may be provided at location 361 affixed to the interior of adriver-side A-pillar of the automobile 300. A sensor may be provided atlocation 362 affixed to the interior of a passenger-side A-pillar of theautomobile 300. A sensor may be provided at other locations in theautomobile 300 beyond those locations just described.

The sensors provided at these one or more locations may be provided asdescribed previously. In some embodiments, a single sensor may be usedat one of the aforementioned locations. In some embodiments, multiplesensors may be used at one of the aforementioned locations. In someembodiments, multiple sensors may be used at more than one of theaforementioned locations. In some embodiments, multiple sensors may beused with each sensor provided at a different one of the aforementionedlocations.

FIG. 4A is an schematic diagram from an overhead view of an automobile400 in which a neural network may be employed according to someembodiments of the present disclosure. Automobile 400 may include adriver's seat 401 and a steering wheel 405, which may be provided asdescribed previously and as used in ordinary automobile environments.

Automobile 400 may include a sensor 460 provided at location 453.Location 453 may be a location for a sensor provided on or under arearview mirror. Therefore, sensor 460 may be provided as a sensor builtinto the a rearview mirror. Alternatively, sensor 460 may be providedseparate from a rearview mirror but affixed to the interior of theautomobile 300, near the rearview mirror.

FIGS. 4B and 4C are illustrations of an interior 470 of an automobileshowing the location of a sensor according to some embodiments of thepresent disclosure. FIG. 4B shows the interior 470 of an automobile suchas that described for automobile 400. In particular, the interior 470includes a steering wheel 405, an instrument cluster 406, and adashboard 408. The interior 470 further includes a rearview mirror 471and a sensor 472. The sensor 472 may be provided as described withrespect to the sensor 460. FIG. 4C shows the interior 470 in furtherdetail, including the rearview mirror 471 and the sensor 472. In theillustrations of FIGS. 4B and 4C, the sensor 472 may be affixed to theinside of the windshield so that the sensor 472 is located just belowthe rearview mirror 471. The sensor 472 may be video camera directed atthe driver's seat.

Referring back to FIG. 4A, use of sensor 460 at location 453 may beadvantageous for embodiments that involve detecting some state of aperson present or absent in driver's seat 401. Location 453 may beadvantageous for such embodiments because sensor 460 mounted at location453 and aimed downward toward the driver's seat 401 may be capable ofsensing the position of substantially all of the driver's body from theknees to the head and including the arms and hands. By sensing theposition of the driver's body or alternatively the presence/absence of adriver, a neural network using the sensor data as input may be able tocategorize the state of the driver into one or more of variouscategories, such as: distracted, not distracted, safe, unsafe, bothhands on the steering wheel, one hand on the steering wheel, no hands onthe steering wheel, looking forward, not looking forward, mobile devicein hand, texting on a mobile device, talking on a mobile device,present, and absent. Other categorizations of the driver's state may bepossible using a neural network and sensor 460.

FIGS. 5A, 5B, 5C, 5D, and 5E are example images of a driver of anautomobile as captured by a sensor in an automobile according to someembodiments of the present disclosure.

Image 501 of FIG. 5A, image 511 of FIG. 5B, and image 521 of FIG. 5C maybe captured by a sensor (e.g., sensors 460, 472) provided built into oraffixed adjacent to a rearview mirror (e.g., rearview mirror 471). Image501 demonstrates that a sensor provided near the rearview mirror maycapture the state of a person in the driver's seat (e.g., distracted,unsafe, texting, one hand on the steering wheel, not looking forward) aswell as the state of a person in the passenger's seat (e.g., present).Image 511 demonstrates that a sensor provided near the rearview mirrormay capture the state of a person in the driver's seat (e.g.,distracted, unsafe, no hands on the steering wheel, not looking forward)as well as the state of a person in the passenger's seat (e.g., notpresent). Image 521 demonstrates that a sensor provided near therearview mirror may capture the state of a person in the driver's seat(e.g., not distracted, unsafe, no hands on the steering wheel, lookingforward, seatbelt buckled, seatbelt not buckled) as well as the state ofa person in the passenger's seat (e.g., not present, seatbelt buckled,seatbelt not buckled).

Images 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, and 562 ofFIG. 5D may be captured by a sensor provided in or near a passenger seat(e.g., passenger seat 302). Each of the images of FIG. 5D may be labeledwith a category based on the state of the driver. For instance, becausethe driver in images 551, 552, 555, 556, 557, 558, 559, 560, 561, and562 is looking forward and has two hands on the steering wheel, each ofthose images may be categorized as “safe driving,” as indicated withlabels 571, 572, 575, 576, 577, 578, 579, 580, 581, and 582. But,because the driver in images 573 and 574 is not looking forward, each ofthose images may be categorized as “unsafe driving,” as indicated withlabels 573 and 574.

Images 583A, 584A, 585A, 586A, 587A, 588A, 589A, 590A, 591A, 592A, 593A,and 594A of FIG. 5E may be captured by a sensor provided near therearview mirror. Each of the images of FIG. 5E may be labeled with acategory based on the state of the driver, as illustrated by labels583B, 584B, 585B, 586B, 587B, 588B, 589B, 590B, 591B, 592B, 593B, and594B, respectively.

FIG. 6A is a diagram of an exemplary neural network structure 600according to some embodiments of the present disclosure. Neural networkstructure 600 includes layers 602, 604, 606, and 608. Neural networkstructure 600 includes connections 612, 614, and 616.

Neural network structure 600 receives input values at input layer 602.Neural network structure 600 then propagates those values throughconnections 612 to layer 604. Each of the connections of connections 612may include a numerical weighting value (e.g., a value between −1 and 1)that is used to modify the original value (e.g., propagatedvalue=original value*weight). The nodes of layer 604 receive thesepropagated values as input. Each node of layer 604 may include afunction that combine the received input values (e.g., summing allreceived inputs). Each node of nodes 604 may further contain one or moreactivation functions that determines when a value will be output on aconnection of connections 614 (e.g., output+1 if the combined value ofthe inputs is >0 and output −1 if the combined value of the inputs is<0, and output 0 if the combined value of the inputs is =0). The outputvalues of the nodes of layer 604 may then be propagated by connections614. Each connection of connections 614 may have a weight value asdescribed with respect to connections 612. The nodes of layer 606 maycombine the received input values and use one or more activationfunctions as described with respect to the nodes of layer 604. Theoutput values of the nodes of layer 606 may then be propagated byconnections 616. Each connection of connections 616 may have a weightvalue as described with respect to connections 612. The nodes of outputlayer 608 may combine the received input values from the connections616. Each node of output layer 608 may correspond to a predefinedcategory for the input values. The combined input values for each nodeof the output layer 608 may determine a category determined for theinput (e.g., the category for the output node that has the largestcombined input values). In this way, neural network structure 600 may beused to determine a category for some input.

The neural network structure 600 may be configured to accuratelydetermine a category for some input through a process called training.For training, numerous inputs are labeled with their correct categoriesby a user or some other actor. The weights for connections 612, 614, and616 may be provided with default and/or random values to start. Theinputs are then provided to the neural network structure 600 throughinput layer 602, and the determined categories for the inputs (e.g.,based on highest combined input values at the nodes of output layer 608)are observed and compared to the correct categories as previouslylabeled. The weights for connections 612, 614, and 616 are thenrepeatedly modified until the neural network structure 600 correctlydetermines the correct categories for all of the inputs, or at least foran acceptable portion of all of the inputs.

When a new input is received without a correct category previouslydetermined, that input can be provided to the neural network structure600 to determine the most likely category for that input.

FIG. 6B is a diagram of an exemplary neural network structure 650according to some embodiments of the present disclosure. In someembodiments, neural network structure 650 may be used to categorize thestate of a driver of an automobile as “safe” or “unsafe” based on animage of the driver. Neural network structure 650 includes layers 652,654, 656, and 658, which may be provided as described with respect tolayers 602, 604, 606, and 608, respectively. Neural network structure650 includes connections 662, 664, and 666, which may be provided asdescribed with respect to connections 612, 614, and 616, respectively.

The input to neural network structure 650 may be an image of the driverof the automobile. The image of the driver (e.g., images 501, 511, 521)may be captured using a sensor (e.g., sensors 460, 472) located near arearview mirror (e.g., rearview mirror 471) of the automobile. The imageof the driver may be converted from its raw captured format (e.g.,8-megapixel color photo) to a compressed format (e.g., 100 pixel×100pixel grayscale image). A numerical value for each pixel (e.g., integergrayscale value between 0 (“black”) and 255 (“white”)) may be inputtedto a separate node of the input layer 652. For example, input node 671may receive the numerical pixel value for the pixel in the topmost andleftmost pixel. Input node 672 may receive the numerical pixel value forthe pixel in the topmost and second-to-leftmost pixel. The numericalpixel values may be assigned to input nodes of layer 652 continuing inthis left-to-right fashion across the topmost row of pixels, thencontinuing with the subsequent rows, until the numerical pixel value forthe bottommost and rightmost pixel is assigned to input node 673.

The output nodes of layer 658 of the neural network structure 650 mayinclude output node 675 and output node 676. Output node 675 maycorrespond to a “safe driving” category, while output node 676 maycorrespond to an “unsafe driving” category.

In order to train the neural network structure 650, driver imagescaptured by an in-automobile sensor may be captured (e.g., as shown inFIGS. 5A, 5B, and 5C), compressed (e.g., as described previously), andlabeled (e.g., as shown in FIG. 5D). The weights for each of theconnections of connections 662, 664, and 666 may be randomly set to avalue between −1 and +1. Each node of layers 654 and 656 may beconfigured to use a combination function (e.g., summation) and anactivation function (e.g., sign of the combined input values) asdescribed previously or otherwise known in the art. The compresseddriver images may then be input to the neural network structure 650(e.g., using the pixel numerical values as inputs to the input layer652). The input values may be propagated through the neural networkstructure 650 as described with respect to the neural network structure600. The category for each input image may be determined as “safedriving” if output node 675 has a combined input values greater than thecombined input values of output node 676. The category for each inputimage may be determined as “unsafe driving” if output node 675 has acombined input values less than or equal to the combined input values ofoutput node 676. These determined categories can be compared to thecorrect categories labeled previously. Using any optimization algorithmknown in the art, the weights of the connections 662, 664, and 666 canbe repeatedly modified until the neural network structure 650 accuratelydetermines the categories for all or at least an acceptable portion ofthe input images.

The neural network structure 650, thus trained, may then be used todetermine the state of the driver (i.e., “safe driving” or “unsafedriving”) at points in the future. This may be accomplished by providingthe neural network structure 650 at a computing device in theautomobile. Then, when the sensor (e.g., sensors 460, 472) located neara rearview mirror (e.g., rearview mirror 471) of the automobile capturesan image of the driver, that image can be compressed and input into thetrained neural network structure 650. The category determined by thetrained neural network structure 650 (i.e., “safe driving” or “unsafedriving”) can then be used as the most likely state of the driver at themoment the driver image was captured.

The foregoing disclosure of neural network structures 600 and 650 wasintended to be exemplary, and neural network structures may be providedin different forms in various embodiments. For example, while neuralnetwork structures 600 and 650 include four layers of nodes, more orfewer layers of nodes may be used in some embodiments. As anotherexample, more output nodes in the output layer may be used in someembodiments (e.g., four nodes representing “safe driving,” “texting,”“touching headunit,” and “talking.” As another example, while neuralnetwork structures 600 and 650 include connections from every node inone layer to every node in the next layer (“fully connected”), fewerconnections may be used in some embodiments. As another example, thenumber of nodes per layer (e.g., more or less than five nodes in layer654) may be different in some embodiments. As another example, whileneural network structures 600 and 650 were described as using weightvalues for each connection and combination and activation functions foreach node, other configurations including more or fewer elements for theneural network structure may be used in some embodiments. As anotherexample, compression of the image captured by the in-automobile sensormay not be used in some embodiments. As another example, conversion tograyscale of the image captured by the in-automobile sensor may not beused in some embodiments. Other modifications of neural networkstructures 600 and 650 in accordance with the present disclosure arepossible in various embodiments.

FIG. 6C is a diagram of an exemplary neural network structure 680according to some embodiments of the present disclosure. In someembodiments, neural network structure 680 may be used to categorize thestate of a driver of an automobile as “safe” or “unsafe” based on animage of the driver. Neural network structure 680 includes layers 682,684, 686, and 688, which may be provided as described with respect tolayers 652, 654, 656, and 658, respectively. Neural network structure680 includes connections 681, 683, and 685, which may be provided asdescribed with respect to connections 662, 664, and 666, respectively.

The input to neural network structure 680 may be an image of the driverof the automobile in addition to other sensor data. The image of thedriver may be provided as input to the neural network structure 680using input node 694, input node 695, and other input nodes of inputlayer 682, as described for neural network structure 650 and input layer652 of FIG. 6B. Additional data may be provided to input nodes 691, 692,and 693. For example, a steering wheel angle value (e.g., degrees ofangular displacement from “wheels-forward” steering wheel position) maybe input into input node 691. An automobile velocity value (e.g.,velocity of the automobile along a roadway) may be input into input node692. A user biometric value (e.g., heartrate of the driver) may be inputinto input node 693. Other image and non-image sensor inputs may be usedin various embodiments.

The output nodes of layer 688 of the neural network structure 680 mayinclude output node 696 and output node 697. Output node 696 maycorrespond to a “ready” category (e.g., the driver is ready to receivecontrol of the automobile as part of a transition from a self-drivingmode to an manual driving mode), while output node 697 may correspond toan “not ready” category (e.g., the driver is not ready to receivecontrol of the automobile as part of a transition from a self-drivingmode to an manual driving mode).

The neural network structure 680 may be trained as described withrespect to neural network structure 650. With neural network structure680, though, a label applied to input data may be applied to a tuple ofinput data: <image, sensor data 1, sensor data 2, sensor data 3>. Thatis, a label provided for the input data may not be specific to just animage provided as input. Rather, the label may be provided as applicableto the entire situation in the automobile as described by the image, thesensor data 1, the sensor data 2, and the sensor data 3. In someembodiments, the image, sensor data 1, and sensor data 2, and sensordata 3 may all be captured in the same automobile at approximately thesame time. As an example, while an image input for a time t1 may showthe driver to have both hands on the wheel and facing forward, if thesteering wheel angle value is 180 degrees at time t1, then the tuple fortime t1 may be labeled “not ready,” reflecting the fact that theautomobile is in a sharp turn and thus the driver may not be ready toreceive control of the automobile. With this clarification, the neuralnetwork structure 680 may be trained using the techniques described withrespect to neural network structure 600 and/or 650.

The neural network structure 680, thus trained, may then be used todetermine the state of the driver (i.e., “ready” or “not ready”) atpoints in the future. This may be accomplished by providing the neuralnetwork structure 680 at a computing device in the automobile. Then,when the sensor (e.g., sensors 460, 472) located near a rearview mirror(e.g., rearview mirror 471) of the automobile captures an image of thedriver, that image and sensor data captured by other sensors can beinput into the trained neural network structure 680. The categorydetermined by the trained neural network structure 680 (i.e., “ready” or“not ready”) can then be used as the most likely state of the driver atthe moment the driver image was captured.

FIG. 7A is a system diagram of a system 700 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 700 includes an automobile 710, a sensor716, a remote computing device 720, a communication link 722, a localtransceiver 724, a local computing device 726, and an automobilecontroller 728.

The automobile 710 may be an automobile as described previously withrespect to automobiles 110, 300, and/or 400. The automobile 710 may beoperated by a driver seated in driver's seat 711. The driver may operatethe automobile 710 using the steering wheel 712 and other controldevices. In some embodiments, the automobile 710 may be configured tooperate in a self-driving or autopilot mode. For example, the automobilemay be configured to cause the automobile 710 to accelerate ordecelerate automatically and without control by the driver (e.g., basedon detection of surrounding hazards). As another example, the automobilemay be configured to steer the automobile 710 (e.g., based on detectinglane lines on a road on which the automobile 710 is travelling). Theoperation of the self-driving or autopilot mode may be controlled by theautomobile controller 728. The automobile controller 728 may beconfigured to control one or more elements in the automobile 710 inorder to control aspects of the operation of the automobile 710. Forexample, the automobile controller 728 may be configured to control: athrottle body, a carburetor, a brake, a transmission, a steeringmechanism, an electronic control module, or other elements.

The sensor 716 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, and/or 472. The sensor716 may be provided at a location 715, such as in a location built intoor in the vicinity of a rearview mirror.

In some embodiments, system 700 may employ a neural network in thefollowing way.

Sensor 716 may sense the position of the driver seated in driver's seat711. Sensor 716 may produce sensor data based on this sensing. Forexample, if sensor 716 is a video camera, the sensor data may be animage of the interior of the automobile 710, including the driver, ifpresent.

The sensor 716 may transmit the image of the interior of the automobile710 to transceiver 724. This transmission may be performed using ashared communications bus, using a dedicated physical cable, using ashort-range wireless link, or in some other way.

The transceiver 724 may transmit the image of the interior of theautomobile 710 to the remote computing device 720 using thecommunication link 722. In some embodiments, the communication link 722may be a dedicated long-range wireless communication link. In someembodiments, the communication link 722 may be a long-range wirelesscommunication link shared with an emergency and navigation service. Insome embodiments, the communication link 722 may be a wiredcommunication link. In some embodiments, the communication link 722 maybe a continuously active communication link. In some embodiments, thecommunication link 722 may be an intermittent communication link.

The remote computing device may save the received image of the interiorof the automobile 710 for later processing. In some embodiments, theremote computing device 720 may transform the image of the interior ofthe automobile 710 (e.g., compressing the image, transforming the imageto grayscale, enhancing brightness, enhancing contrast). In someembodiments, the sensor 716, the transceiver 724, or some othercomputing device located in the automobile 710 (e.g., local computingdevice 726) may transform the image of the interior of the automobile710 (e.g., compressing the image, transforming the image to grayscale,enhancing brightness, enhancing contrast) prior to transmitting theimage of the interior of the automobile 710 to the remote computingdevice 720.

The sensor 716, the transceiver 724, and the remote computing device 720may repeat this process of image capture, transmission, and storagenumerous times until a large quantity of images are stored by the remotecomputing device 720. In some embodiments, the large quantity of imagesmay be 100 images. In some embodiments, the large quantity of images maybe 1,000 images. In some embodiments, the large quantity of images maybe 10,000 images. In some embodiments, the large quantity of images maybe 100,000 images.

One or more users may label the images based on predefined categories(e.g., “safe driving” and “unsafe” driving). The remote computing device720 may store these category labels with the images to which theypertain. In some embodiments, a user located in the automobile 710(e.g., the driver, a passenger) may perform the labelling prior totransmitting the corresponding image to the remote computing device 720.The one or more users may label the images while the remote computingdevice 720 is still waiting to finish storage a large quantity ofimages, after completing storage of a large quantity of images, or atsome other time. In some embodiments, the one or more users may labelthe images using a user interface (not pictured) communicatively coupledto the remote computing device 720 (e.g., the user interface of a laptopwith the laptop communicating with the remote computing device 720).

Once a large quantity of images and associated category labels arestored by the remote computing device 720, the remote computing device720 may train a neural network structure (e.g., neural networkstructures 600, 650) based on the images and corresponding labels. Forexample, if the images are images of the interior of the vehicle 710including the position of the driver's upper legs, torso, arms, hands,and head, and the category labels are each selected from “safe driving”and “unsafe driving,” the remote computing device 720 may train a neuralnetwork structure to accurately categorize the state of the driver as“safe driving” or “unsafe driving” based on the driver's body position.

Once the remote computing device 720 trains the neural networkstructure, the remote computing device 720 may transmit parametersdefining the neural network structure (also referred to herein as“neural network configuration parameters”) (e.g., specification ofnodes, specification of combination functions for nodes, specificationof activation functions for nodes, specification of connections, and/orspecification of weights for connections) to the local transceiver 724using the communication link 722. In some embodiments, the remotecomputing device 720 may transmit the parameters defining the neuralnetwork structure to the local transceiver 724 using a communicationlink other than communication link 722.

The local transceiver 724 may transmit the parameters defining theneural network structure to the local computing device 726. The localcomputing device 726 may be configured to receive the parametersdefining the neural network structure from the local transceiver. Thelocal computing device 726 may also be configured to receive an image ofthe interior of the automobile 710 from the sensor 716. This receptionfrom the local transceiver 724 and the sensor 716 may be performed usingany of a variety of communication mediums, such as those describedpreviously for communication between the sensor 716 and the localtransceiver 724.

The local computing device 726 may be configured to create thefully-trained neural network structure (e.g., recreate the structure inlocal memory) based on the parameters defining the neural networkstructure. Once the local computing device 726 receives the parametersdefining the neural network structure, the local computing device 726may provide an image of the interior of the automobile 710 as input tothe neural network structure in real-time. The local computing device726 may thereby determine a category for the an image of the interior ofthe automobile 710 (e.g., “safe driving” or “unsafe driving”) whichindicates the most likely state of the driver of the automobile 710 whenthe image was captured. If the images of the interior of the automobile710 is provided to the local computing device 726 in substantiallyreal-time, then the local computing device 726 may be able to determinewhether the driver is in an unsafe driving state or a safe driving stateat substantially the present time.

The local computing device 726 may transmit the determined state of thedriver (e.g., “safe driving” or “unsafe driving”) to the automobilecontroller 728. This transmission to the automobile controller 728 maybe performed using any of a variety of communication mediums, such asthose described previously for communication between the sensor 716 andthe local transceiver 724.

The automobile controller 728 may be configured to control whether theautomobile 710 automatically transitions from a self-driving orautopilot mode to a manual mode based on the state of the driverdetermination received from the local computing device 726. For example,the automobile controller 728 may be configured to operate in aself-driving or autopilot mode while on an interstate. But, theautomobile controller 728 may be configured to transition to a manualmode (e.g., with full control of the vehicle by the driver) when theautomobile 710 exits the interstate. However, the automobile controller728 may determine to restrict transition from a self-driving orautopilot mode to a manual mode even if the automobile 710 is exiting aninterstate, if the local computing device 726 determines that the driveris presently in an “unsafe driving” state.

The automobile controller 728 may be configured to control whether theautomobile 710 automatically transitions from a manual mode to aself-driving or autopilot mode based on the state of the driverdetermination received from the local computing device 726. For example,the automobile controller 728 may be configured to automaticallytransition to self-driving or autopilot mode that includes aself-steering operation if the automobile 710 is on an interstate andthe local computing device 726 determines that the driver is presentlyin an “unsafe driving” state.

In some embodiments of the present disclosure, different categories thanthose just described for system 700 may be used to categorize a state ofthe driver. For example, the system 700 may include a neural networkstructure trained to determine a driver state as one of the following:“safe driving,” “texting,” “talking,” and “headunit.” In suchembodiments, the images used by the remote computing device 720 to trainthe neural network structure may be labeled with these categories sothat the trained neural network structure may accurately determine thedriver's state from these categories. In such embodiments, theautomobile controller 728 may be configured to restrict a transitionfrom a self-driving mode to a manual mode for only some of the driverstates (e.g., for “texting” and “talking” but not “safe driving” and“headunit”).

Other categories of driver state are possible in various embodiments ofthe present disclosure. For example, the system 700 may include a neuralnetwork structure trained to determine a driver state as one of thefollowing: “looking forward,” “looking down,” “looking at mobiledevice,” “looking at front passenger seat,” “looking at rear passengerseat,” “looking in rearview mirror,” “looking in left side-view mirror,”“looking in right side-view mirror,” “looking at left blindspot,” and“looking at right blindspot.” Other categories of driver state may beused in accordance with various embodiments of the present disclosure.

In some embodiments of the present disclosure, the images used by theremote computing device 720 to train the neural network structure may becaptured in one or more first automobiles, while the trained neuralnetwork structure may be used by a local computing device 726 in asecond, different automobile 710. For example, the images captured fortraining of the neural network structure may be captured during a setupperiod (e.g., by sensors deployed in tens or even hundreds ofautomobiles, prior to installation of a local computing device capableof using a neural network structure in an automobile). These imagescaptured during the setup period may be captured so as to cover a widerange of drivers (e.g., tall people, short people, thin people, thickpeople, men, women). This may allow the remote computing device 720 totrain a neural network structure that can accurately determine adriver's state regardless of the type of driver. That is, the remotecomputing device 720 may be capable of training a generic neural networkstructure for use with many different drivers. A generic neural networkstructure so trained may then be provided pre-installed on the localcomputing device 726 when the local computing device 726 is firstinstalled in the automobile 710 and/or when the automobile 710 is firstsold to an end user. Other approaches to training and then implementingthe neural network structure in an automotive environment may be used inaccordance with various embodiments of the present disclosure.

In some embodiments of the present disclosure, the remote computingdevice 720 may train more than one neural network structure tocategorize a driver state. For example, a first set of driver images maybe captured of numerous drivers (e.g., a sensor 716 installed in each often cars). A second set of driver images may be captured of numerousother drivers (e.g., a sensor 716 installed in each of ten other cars).The first set of driver images and the second set of driver images maybe labeled (e.g., as “safe driving” and “unsafe driving”). The remotecomputing device 720 may train a first neural network structure usingthe first set of driver images and second neural network structure usingthe second set of driver images. The remote computing device 720 maythen transmit the first neural network structure, the second neuralnetwork structure, or both to the local transceiver 724. The localcomputing device may then use either or both of the neural networkstructures in real-time to determine a state of the driver.

The aforementioned training of multiple neural network structures toperform the same categorization may be advantageous in a number ofsituations. For example, the first set of driver images may be capturedof drivers sharing similar first physical appearances (e.g., tall peoplewith short hair), while the second set of driver images may be capturedof drivers sharing similar second physical appearances different fromthose of the first set of drivers (e.g., short people with long hair).Because the neural network structures are trained on images in someembodiments, these differences in physical appearances can affect theaccuracy of the neural network structures' categorization. Therefore,the first neural network structure may be highly accurate for the firstphysical appearances but less so for the second physical appearances(e.g., highly accurate at categorizing as “safe driving” or “unsafedriving” a tall driver with short hair, but not so for a short driverwith long hair), and vice versa for the second neural network structure.In such embodiments, the remote computing device 720 may only transmitand/or the local transceiver 724 may only receive the neural networkstructure that is best suited for the driver of the automobile 710(e.g., if the driver of the automobile 710 is a tall person with shorthair, then only transmit/receive the first neural network structure).Alternatively, the remote computing device 720 may transmit both neuralnetwork structures to the automobile 710, and the two neural networkstructures may be evaluated in the automobile 710 and with its driver inorder to determine which is more accurate (e.g., which more frequentlydetermines the correct category, or which determines the correctcategory with a higher confidence level). In other embodiments, morethan two neural network structures may be trained to perform the samecategorization.

In some embodiments of the present disclosure, the remote computingdevice 720 may provide the parameters defining the neural networkstructure to the local transceiver 724 by recording the parameters in ablockchain. For example the remote computing device 720 may record theparameters defining the neural network structure in a local block (i.e.,local to the remote computing device 720) of the blockchain. Theblockchain may then promulgate the local block to other computingdevices. The local transceiver 724 may then read the parameters definingthe neural network structure from one of these blocks located on one ofthe other computing devices. The use of a blockchain to transmit theparameters defining the neural network structure to the localtransceiver 724 may be advantageous because it may guarantee theauthenticity of those parameters (e.g., guaranteeing that a third-partyhas not tampered with the parameters for malicious purposes orotherwise).

In some embodiments of the present disclosure, the remote computingdevice 720 may train the neural network structure using a stereovisionapproach. For example, the system 700 may use the sensor 716 located atlocation 715 near the rearview mirror, and a second sensor affixed tothe driver-side A-frame of the automobile 710 (e.g., at location 361 ofFIG. 3). In such embodiments, both the images captured by the sensor 716and the images captured by the A-frame sensor may be transmitted to theremote computing device 720. The remote computing device 720 may thencombine each image captured by sensor 716 with a corresponding imagefrom the A-frame sensor. This correspondence relationship may bedetermined as a rearview mirror image and an A-frame image that wereboth captured at the same time. The combining of the images may includeoverlaying one of the images over part of the other image (e.g., basedon overlapping fields of view captured by the images) so as to form asingle image. The remote computing device 720 may then use this combinedstereovision image to train the neural network structure. The localcomputing device 726 may then perform the same image combining so as toprovide a combined sterevision image as input to the neural networkstructure in real-time. This stereovision image approach may beadvantageous as capturing a more comprehensive view of the driver. Insome embodiments, more than two images may be combined to form thestereovision image. In some embodiments, images from sensors not locatedat a rearview mirror and/or an A-frame may be combined into astereovision image. In some embodiments, other computing devices mayperform the combining (e.g., the sensor 716). In some embodiments, thecombining may be performed using techniques other than the overlayingtechnique just described (e.g., cropping and adding).

In some embodiments of the present disclosure, the system 700 mayperform outlier reporting. For example, when applying a driver image asinput to the neural network structure, the local computing device 726may be unable to determine the state of the driver with a high degree ofconfidence (e.g., the neural network may report a 48% confidence thatthe driver is “unsafe driving” and a 52% confidence that the driver is“safe driving”). In such cases, the local computing device 726 may storethe images that could not be categorized to a high degree of confidence.In some embodiments, the local computing device 726 may later transmitthese outlier images to the remote computing device 720 (e.g., by way ofthe local transceiver 724). The remote computing device may thenassociate labels with these outlier images (e.g., based on a userinput), combine these outlier images with other images captured by thesensor 716, a train an update neural network structure. The remotecomputing device 720 may then provide this update neural networkstructure (e.g., by transmission of parameters defining the neuralnetwork structure) to the local computing device 726 (e.g., by way ofthe local transceiver 724) for use. Such an outlier reporting andupdating approach may be advantageous as allowing the system 700 toupdate the neural network structure over time in order to improve theaccuracy of its ability to categorize the driver's state.

FIG. 7B is a system diagram of a system 750 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 750 includes an automobile 760, a sensor765, a sensor 766, a sensor 767, a sensor 768, a sensor 769, a remotecomputing device 770, a communication link 772, a local transceiver 774,a local computing device 776, and an automobile controller 778.

The automobile 760 may be an automobile as described previously withrespect to automobiles 110, 300, and/or 400. The automobile 760 may beoperated by a driver seated in driver's seat 761. The driver may operatethe automobile 760 using the steering wheel 762 and other controldevices. In some embodiments, the automobile 760 may be configured tooperate in a self-driving or autopilot mode. For example, the automobilemay be configured to cause the automobile 760 to accelerate ordecelerate automatically and without control by the driver (e.g., basedon detection of surrounding hazards). As another example, the automobilemay be configured to steer the automobile 760 (e.g., based on detectinglane lines on a road on which the automobile 760 is travelling). Theoperation of the self-driving or autopilot mode may be controlled by theautomobile controller 778. The automobile controller 778 may beconfigured to control one or more elements in the automobile 760 inorder to control aspects of the operation of the automobile 760. Forexample, the automobile controller 778 may be configured to control: athrottle body, a carburetor, a brake, a transmission, a steeringmechanism, an electronic control module, or other elements.

The sensor 765 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, and/or 472. The sensor765 may be an image capture device (e.g., a camera) in a location builtinto or in the vicinity of a driver's-side A-pillar.

The sensor 766 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, and/or 472. The sensor766 may be an image capture device (e.g., a camera) in a location builtinto or in the vicinity of a rearview mirror.

The sensor 767 may be a sensor capable of detecting a biometric state ofthe driver of the automobile 760. For example, the sensor 767 may be asensor configured to detect the heartrate of the driver. In such cases,the sensor 767 may be provided as a wrist watch, a chest strap, anelectric circuit in the steering wheel with contact points for each ofthe driver's hands, etc. As another example, the sensor 767 may be asensor configured to detect the blood pressure of the driver. In suchcases, the sensor 767 may be provided as a wrist watch, an arm strap,etc. The sensor 767 may be provided to detect other biometric states ofthe driver of the automobile 760 in accordance with various embodimentsof the present disclosure.

The sensor 768 may be a sensor capable of detecting a state of acomponent of the automobile 760. For example, the sensor 768 may becapable of detecting an angular displacement of the steering wheel 762(e.g., degrees of rotation from a “wheels forward” position). The sensor768 may be provided built into or attached to the steering wheel 762. Insuch cases, the sensor 768 may be provided as a gyroscope, apotentiometer, an optical sensor (e.g., detecting markings on a mountingpoint for the steering wheel 762), or in some other form. The sensor 768may be provided to detect other states of components of the automobile760 in accordance with various embodiments of the present disclosure.

The sensor 769 may be a sensor capable of detecting a state of theautomobile 760. For example, the sensor 769 may be capable of detectinga velocity of the automobile 760 (e.g., a velocity along a roadway). Insuch cases, the sensor 760 may be provided as a Global PositioningSystem receiver and configured to detect the current velocity based onthe rate of change of the position of the automobile 760, an opticalsensor attached to a tire of the automobile 760 and configured to detectthe current velocity based on the rate of revolution of the tire, etc.The sensor 769 may be provided to detect other states of the automobile760 in accordance with various embodiments of the present disclosure.

In some embodiments, system 750 may employ a neural network in thefollowing way.

Sensor 765 may sense the position of the driver seated in driver's seat761. Sensor 765 may produce sensor data based on this sensing. Forexample, if sensor 765 is a video camera, the sensor data may be animage of the interior of the automobile 760, including the driver, ifpresent. In some embodiments, the sensor 765 may be provided as a cameramounted on a driver-side A-pillar of the automobile 760. In suchembodiments, the sensor 765 may be aimed at the lower portion of thedriver's body, such as the arms, hands, torso, and upper legs.

The sensor 765 may transmit the image of the interior of the automobile710 to transceiver 774. This transmission may be performed using ashared communications bus, using a dedicated physical cable, using ashort-range wireless link, or in some other way.

Sensor 766 may sense the position of the driver seated in driver's seat761. Sensor 766 may produce sensor data based on this sensing. Forexample, if sensor 766 is a video camera, the sensor data may be animage of the interior of the automobile 760, including the driver, ifpresent. In some embodiments, the sensor 766 may be provided as a cameramounted in the vicinity of a rearview mirror of the automobile 760. Insuch embodiments, the sensor 766 may be aimed at the upper portion ofthe driver's body, such as neck and head.

The sensor 766 may transmit the image of the interior of the automobile710 to transceiver 774. This transmission may be performed using ashared communications bus, using a dedicated physical cable, using ashort-range wireless link, or in some other way.

Sensor 767 may sense a biometric state of the driver seated in driver'sseat 761. Sensor 767 may produce sensor data based on this sensing. Forexample, if sensor 767 is a wristwatch with a built-in heartratemonitor, the sensor data may be an electronic signal identifying anumber of beats per minute for the heart of the driver. The sensor 767may transmit the sensor data of the biometric state of the driver totransceiver 774. This transmission may be performed using a sharedcommunications bus, using a dedicated physical cable, using ashort-range wireless link, using a Bluetooth Low Energy connection, orin some other way.

Sensor 768 may sense a state of a component of the automobile 760.Sensor 768 may produce sensor data based on this sensing. For example,if sensor 768 is a gyroscope built into the steering wheel 762, thesensor data may be an electronic signal identifying an angulardisplacement of the steering wheel 762 from a “wheels forward” position.The sensor 768 may transmit the sensor data of the state of thecomponent of the automobile 760 to transceiver 774. This transmissionmay be performed using a shared communications bus, using a dedicatedphysical cable, using a short-range wireless link, or in some other way.

Sensor 769 may sense a state of the automobile 760. Sensor 769 mayproduce sensor data based on this sensing. For example, if sensor 769 isa Global Positioning System receiver, the sensor data may be anelectronic signal identifying a velocity of the automobile 760. Thesensor 769 may transmit the sensor data of the state of the automobile760 to transceiver 774. This transmission may be performed using ashared communications bus, using a dedicated physical cable, using ashort-range wireless link, or in some other way.

The transceiver 774 may transmit sensor data received from sensor 765,766, 767, 768, and 769 to the remote computing device 770 using thecommunication link 772. In some embodiments, the communication link 772may be a dedicated long-range wireless communication link. In someembodiments, the communication link 772 may be a long-range wirelesscommunication link shared with an emergency and navigation service. Insome embodiments, the communication link 772 may be a wiredcommunication link. In some embodiments, the communication link 772 maybe a continuously active communication link. In some embodiments, thecommunication link 772 may be an intermittent communication link.

The remote computing device 770 may save the received sensor data forlater processing. In some embodiments, the remote computing device 770may transform the sensor data (e.g., compressing an image, transformingan image to grayscale, enhancing brightness an image, enhancing contrastof an image, combining images from several sensor into a singlestereovision image). In some embodiments, the sensor 765, the sensor766, the sensor 767, the sensor 768, the sensor 769, the transceiver774, or some other computing device located in the automobile 760 (e.g.,local computing device 776) may transform the sensor data (e.g.,compressing an image, transforming an image to grayscale, enhancingbrightness of an image, enhancing contrast of an image, combining imagesfrom several sensor into a single stereovision image) prior totransmitting the sensor data to the remote computing device 770.

The remote computing device 770 may associate multiple elements ofreceived sensor data into tuples. For example, the remote computingdevice 770 may associate an image captured by sensor 765 with an imagecaptured by sensor 766, a biometric state captured by sensor 767, astate of a component of the automobile 760 captured by sensor 768, and astate of the automobile 760 captured by sensor 769. The remote computingdevice 770 may association sensor data into tuples using a timestampassociated with each element of sensor data. For example, the remotecomputing device 770 may associate an image captured by sensor 765 attime t1 with the sensor data captured by the other sensors at timesclosest to time t1 (e.g., the image captured by sensor 766 closest totime t1, the biometric state captured by sensor 767 closest to time t1,etc.). In some embodiments, the sensor 765, the sensor 766, the sensor767, the sensor 768, the sensor 769, the transceiver 774, or some othercomputing device located in the automobile 760 (e.g., local computingdevice 776) may associate the sensor data prior to transmitting thesensor data to the remote computing device 770.

The sensor 765, the sensor 766, the sensor 767, the sensor 768, thesensor 769, the transceiver 774, and the remote computing device 770 mayrepeat this process of sensor data capture, transmission, and storagenumerous times until a large quantity of sensor data are stored by theremote computing device 770. In some embodiments, the large quantity ofsensor data may be 100 tuples of associated sensor data. In someembodiments, the large quantity of images may be 1,000 tuples ofassociated sensor data. In some embodiments, the large quantity ofimages may be 10,000 tuples of associated sensor data. In someembodiments, the large quantity of images may be 100,000 tuples ofassociated sensor data.

One or more users may label the tuples of sensor data based onpredefined categories (e.g., “ready” and “not ready” driving). Theremote computing device 770 may store these category labels with thetuples of sensor data to which they pertain. In some embodiments, a userlocated in the automobile 760 (e.g., the driver, a passenger) mayperform the labelling prior to transmitting the corresponding tuple ofsensor data to the remote computing device 770. The one or more usersmay label the tuples of sensor data while the remote computing device770 is still waiting to finish storage a large quantity of sensor data,after completing storage of a large quantity of sensor data, or at someother time. In some embodiments, the one or more users may label thetuples of sensor data using a user interface (not pictured)communicatively coupled to the remote computing device 770 (e.g., theuser interface of a laptop with the laptop communicating with the remotecomputing device 770).

Once a large quantity of tuples of sensor data and associated categorylabels are stored by the remote computing device 770, the remotecomputing device 770 may train a neural network structure (e.g., neuralnetwork structures 600, 650, 680) based on the tuples of sensor data andcorresponding labels. For example, if the tuples of sensor data includeare images of the interior of the vehicle 760, the state of the driver,the state of a component of the automobile 760, and the state of theautomobile 760, and the category labels are each selected from “ready”and “not ready,” the remote computing device 770 may train a neuralnetwork structure to accurately categorize the state of the driver as“ready” (e.g., to perform manual operation of the automobile 760, as atransition from a self-driving mode) or “not ready” (e.g., to performmanual operation of the automobile 760, as a transition from aself-driving mode) based on the driver's body position, the driver'sbiometric state, the state of the component of the automobile 760, andthe state of the automobile 760.

Once the remote computing device 770 trains the neural networkstructure, the remote computing device 770 may transmit parametersdefining the neural network structure (also referred to herein as“neural network configuration parameters”) (e.g., specification ofnodes, specification of combination functions for nodes, specificationof activation functions for nodes, specification of connections, and/orspecification of weights for connections) to the local transceiver 774using the communication link 772. In some embodiments, the remotecomputing device 770 may transmit the parameters defining the neuralnetwork structure to the local transceiver 774 using a communicationlink other than communication link 772.

The local transceiver 774 may transmit the parameters defining theneural network structure to the local computing device 776. The localcomputing device 776 may be configured to receive the parametersdefining the neural network structure from the local transceiver 774.The local computing device 776 may also be configured to receive animage of the interior of the automobile 760 from the sensor 765, animage of the interior of the automobile 760 from the sensor 766, abiometric state of the driver from the sensor 767, a state of acomponent of the automobile 760 from the sensor 768, and a state of theautomobile 760 from the sensor 769. This reception from the localtransceiver 774, the sensor 765, the sensor 766, the sensor 767, thesensor 768, and the sensor 769 may be performed using any of a varietyof communication mediums, such as those described previously forcommunication between the sensor 765 and the local transceiver 774.

The local computing device 776 may be configured to create thefully-trained neural network structure (e.g., recreate the structure inlocal memory) based on the parameters defining the neural networkstructure. Once the local computing device 776 receives the parametersdefining the neural network structure, the local computing device 776may provide the sensor data received from each of the sensors 765, 766,767, 768, and 769 as input to the neural network structure in real-time.The local computing device 776 may thereby determine a category for theinput sensor data (e.g., “ready” or “not ready”) which indicates themost likely state of the driver of the automobile 760 when the sensordata was captured. If the sensor data is provided to the local computingdevice 776 in substantially real-time, then the local computing device776 may be able to determine whether the driver is in a ready state or anot ready state at substantially the present time.

The local computing device 776 may transmit the determined state of thedriver (e.g., “ready” or “not ready”) to the automobile controller 778.This transmission to the automobile controller 778 may be performedusing any of a variety of communication mediums, such as those describedpreviously for communication between the sensor 765 and the localtransceiver 774.

The automobile controller 778 may be configured to control whether theautomobile 760 automatically transitions from a self-driving orautopilot mode to a manual mode based on the state of the driverdetermination received from the local computing device 776. For example,the automobile controller 778 may be configured to operate in aself-driving or autopilot mode while on an interstate. But, theautomobile controller 778 may be configured to transition to a manualmode (e.g., with full control of the vehicle by the driver) when theautomobile 760 exits the interstate. However, the automobile controller778 may determine to restrict transition from a self-driving orautopilot mode to a manual mode even if the automobile 760 is exiting aninterstate, if the local computing device 726 determines that the driveris presently in a “not ready” state.

The automobile controller 778 may be configured to use images, driverbiometric state information, automobile state information, and/orautomobile component state information to control operation of theautomobile 760 in other ways in accordance with various embodiments ofthe present disclosure.

FIG. 8 is a diagram of a sensor device 800 according to some embodimentsof the present disclosure. The sensor device 800 may be provided as asensor (e.g., sensors 232, 234, 460, 472) in some embodiments of thepresent disclosure. The sensor device 800 includes a lens 802, an imagesensor 806, a memory 810, a processor 812, a power input 814, and atransceiver 816.

The sensor device 800 may use the lens 802 and image sensor 806 tocapture an image of the interior of an automobile (e.g., automobiles110, 300, 400, 710). The lens 802 may be coupled to the image sensor bycoupling 804 (e.g., a physical attachment) and used to focus light onthe image sensor 806. The image sensor 806 may be configured to convertabsorbed light into an analog or digital signal reflecting the absorbedlight.

The image sensor 806 may be coupled to the memory 810, processor 812,power input 814, and transceiver 816 by coupling 808 (e.g., a wiredcommunications bus). The memory 810 may be configured to temporarilystore information pertaining to the image captured by the image sensor806, either before or after processing by the processor 812. Theprocessor 812 may be configured to process information stored by thememory 810 and/or generated by the image sensor 806 (e.g., imagecompression, analog to digital conversion).

The power input 814 may be configured to provide electric power to theimage sensor 806, memory 810, processor 812, and/or transceiver 816. Insome embodiments, the power input 814 may include an electrical couplingto each of the image sensor 806, memory 810, processor 812, and/ortransceiver 816 apart from the coupling 808.

The transceiver 816 may be configured to transmit information pertainingto the image captured by the image sensor 806 to one or more externaldevices, such as a local transceiver (e.g., local transceiver 724) or alocal computing device (e.g., local computing device 726). In someembodiments, the transceiver 816 may be configured to transmitinformation pertaining to information pertaining to the image capturedby the image sensor 806 after the information has been processed by theprocessor 812. In some embodiments, the transceiver 816 may include anetwork adaptor configured to format information pertaining to the imagecaptured by the image sensor 806 across a data communications network.

FIG. 9 is a diagram of a local transceiver according to some embodimentsof the present disclosure. The local transceiver 900 may be provided asa local transceiver (e.g., local transceiver 724) in some embodiments ofthe present disclosure. The local transceiver 900 includes a transceiver902, a coupling 904, a memory 906, a processor 908, a power input 910,and a transceiver 912.

The transceiver 902, memory 906, processor 908, power input 910, andtransceiver 912 may be coupled together by coupling 904 (e.g., a wiredcommunications bus). In some embodiments, only a subset of transceiver902, memory 906, processor 908, power input 910, and transceiver 912 maybe coupled together by coupling 904.

The transceiver 902 may be configured to communicate with other devicesco-located in an automobile (e.g., sensor 716, local computing device726). For example, the transceiver 902 may be configured to receiveinformation pertaining to an image captured by a sensor device (e.g.,sensor device 800) co-located in an automobile. As another example, thetransceiver 902 may be configured to transmit neural networkconfiguration parameters to a computing device (e.g., local computingdevice 726, 1100) co-located in an automobile.

The memory 906 may be configured to temporarily store informationpertaining to image information received by the transceiver 902 orneural network structure information received by the transceiver 912,either before or after processing by the processor 908. For example, thememory 906 may be configured to buffer image information after receptionby transceiver 902 and prior to transmission by transceiver 912. Asanother example, the memory 906 may be configured to buffer neuralnetwork configuration parameters after reception by transceiver 912 andbefore transmission by transceiver 902.

The processor 908 may be configured to process information stored by thememory 906, received by the transceiver 902, and/or received by thetransceiver 912. The processor 908 may be configured to perform variousdata processing tasks (e.g., image compression, data serialization).

The power input 910 may be configured to provide electric power to thetransceiver 902, memory 906, processor 908, and transceiver 912. In someembodiments, the power input 910 may include an electrical coupling toeach of the transceiver 902, memory 906, processor 908, and transceiver912 apart from the coupling 904.

The transceiver 912 may be configured to communicate with a remotecomputing device (e.g., remote computing devices 130, 720, 1000) thatare not co-located in an automobile with the transceiver device 900. Forexample, the transceiver 912 may be configured to transmit informationpertaining to an image captured by a sensor device (e.g., sensor device800) co-located in an automobile to a remote computing device. Asanother example, the transceiver 912 may be configured to receive neuralnetwork configuration parameters from a remote computing device. Thetransceiver 912 may be configured to communicate with the remotecomputing device using a communication link (e.g., communication links140, 722). In some embodiments, the communication link may be a cellularcommunication link, in which case the transceiver 912 may include acellular communication (e.g., CDMA2000, GSM, 4G LTE) transceiver. Insome embodiments, the communication link may be a IEEE 802.11 Wi-Filink, in which case the transceiver 912 may include a Wi-Fi transceiver.In some embodiments, the communication link may be a wired link, inwhich case the transceiver 912 may include a wired (e.g., Ethernet, USB)transceiver. In some embodiments, the communication link may involve aphysical transfer of storage media (e.g., transfer of a flash drive). Insome embodiments, the communication link may be a dedicated cellularlink (e.g., a link to a cellular radio access network providedspecifically for the system 700.) In some embodiments, the communicationlink may be a shared cellular link (e.g., a link to a cellular radioaccess network shared with other services in the automobileenvironment.) Other forms of communications links and communicationstransceivers may be used in various embodiments of the presentdisclosure.

FIG. 10 is a diagram of a remote computing device 1000 according to someembodiments of the present disclosure. The remote computing device 1000may be provided as a remote computing device (e.g., remote computingdevices 130, 720) in some embodiments of the present disclosure. Theremote computing device 1000 includes a transceiver 1002, a coupling1004, a memory 1006, a processor 1008, a power input 1010, and a datastore 1012.

The transceiver 1002, memory 1006, processor 1008, power input 1010, anddata store 1012 may be coupled together by coupling 1004 (e.g., a wiredcommunications bus). In some embodiments, only a subset of transceiver1002, memory 1006, processor 1008, power input 1010, and data store 1012may be coupled together by coupling 1004.

The transceiver 1002 may be configured to communicate with a localtransceiver (e.g., local transceiver 724, 900) that is located in anautomobile and remote from the remote computing device 1000. Forexample, the transceiver 1002 may be configured to receive informationpertaining to an image captured by a sensor device (e.g., sensor device800) located in an automobile from a local computing device co-locatedwith the sensor device in the automobile. As another example, thetransceiver 1002 may be configured to transmit neural networkconfiguration parameters from to the local computing device. Thetransceiver 1002 may be configured to communicate with the localcomputing device using a communication link (e.g., communication links140, 722). In some embodiments, the communication link may be a cellularcommunication link, in which case the transceiver 1002 may include acellular communication (e.g., CDMA2000, GSM, 4G LTE) transceiver. Insome embodiments, the communication link may be a IEEE 802.11 Wi-Filink, in which case the transceiver 1002 may include a Wi-Fitransceiver. In some embodiments, the communication link may be a wiredlink, in which case the transceiver 1002 may include a wired (e.g.,Ethernet, USB) transceiver. In some embodiments, the communication linkmay involve a physical transfer of storage media (e.g., transfer of aflash drive). Other forms of communications links and communicationstransceivers may be used in various embodiments of the presentdisclosure.

The memory 1006 may be configured to temporarily store informationpertaining to image information received by the transceiver 1002 orneural network structure information prepared for transmission by thetransceiver 1002, either before or after processing by the processor1008. For example, the memory 1006 may be configured to buffer imageinformation after reception by transceiver 1002 and prior to storage indata store 1012. As another example, the memory 1006 may be configuredto buffer neural network configuration parameters after generation bythe process 1008 and before transmission by transceiver 1002. In someembodiments, the memory 1006 may be further configured to store neuralnetwork configuration parameters used by the processor 1008 duringtraining of a neural network structure (e.g., neural network structures600, 650).

The processor 1008 may be configured to process information stored bythe memory 1006, received by the transceiver 1002, and/or stored by thedata store 1012. The processor 1008 may be configured to perform variousdata processing tasks. For example, the processor 1008 may be configuredto initialize default neural network configuration parameters prior totraining a neural network structure (e.g., neural network structures600, 650). As another example, the processor 1008 may be configured toprovide images stored by the data store 1012 as inputs to the neuralnetwork structure. As another example, the processor 1008 may beconfigured to repeatedly modify the neural network configurationparameters so that the neural network structure accurately determinesthe categories of the images stored by the data store 1012 based oncategory labels stored by data store 1012. In some embodiments, theprocessor 1008 may be configured to train a neural network structure. Insome embodiments, the processor 1008 may be provided as a singleprocessor as part of the remote computing device 1000. In someembodiments, the processor 1008 may be provided as multiple processorsas part of the remote computing device 1000. In some embodiments, theprocessor 1008 may be provided as multiple cores of one or moreprocessors as part of the remote computing device 1000. In someembodiments, the processor 1008 may be provided as multiple processorsas part of more than one remote computing device 1000. Other forms ofprocessor configurations may be used in various embodiments of thepresent disclosure.

The power input 1010 may be configured to provide electric power to thetransceiver 1002, memory 1006, processor 1008, and data store 1012. Insome embodiments, the power input 1010 may include an electricalcoupling to each of the transceiver 1002, memory 1006, processor 1008,and data store 1012 apart from the coupling 1004.

The data store 1012 may be configured to store image informationpertaining to an image captured by a sensor device in an automobile,category labels pertaining to one or more images stored by the datastore 1012, and/or neural network configuration parameters generated bythe processor 1008. In some embodiments, the data store 1012 may beprovided as a single storage device as part of the remote computingdevice 1000. In some embodiments, the data store 1012 may be provided asmultiple storage devices as part of the remote computing device 1000. Insome embodiments, the data store 1012 may be provided as multiplestorage devices as part of more than one remote computing devices 1000.Other forms of storage configurations may be used in various embodimentsof the present disclosure.

FIG. 11 is a diagram of a local computing device according to someembodiments of the present disclosure. The local computing device 1100may be provided as a local computing device (e.g., local computingdevice 726) in some embodiments of the present disclosure. The localcomputing device 1100 includes a transceiver 1102, a coupling 1104, amemory 1106, a processor 1108, a power input 1110, and a data store1112.

The transceiver 1102, memory 1106, processor 1108, power input 1110, anddata store 1112 may be coupled together by coupling 1104 (e.g., a wiredcommunications bus). In some embodiments, only a subset of transceiver1102, memory 1106, processor 1108, power input 1110, and data store 1112may be coupled together by coupling 1104.

The transceiver 1102 may be configured to communicate with other devicesco-located in an automobile (e.g., sensor 716, local transceiver 724,automobile controller 728). For example, the transceiver 1102 may beconfigured to receive information pertaining to an image captured by asensor device (e.g., sensor device 800) co-located in an automobile. Asanother example, the transceiver 1102 may be configured to receiveneural network configuration parameters from a local transceiver device(e.g., local transceiver 724, 900) co-located in an automobile. Asanother example, the transceiver 1102 may be configured to transmit toan automobile controller (e.g., automobile controller 728, 1200) acategory determined to apply to an image based on input of the image toa neural network structure.

The memory 1106 may be configured to temporarily store informationpertaining to image information received by the transceiver 1102 orneural network structure information prepared by the processor 1108. Forexample, the memory 1106 may be configured to buffer image informationafter reception by transceiver 1102. As another example, the memory 1106may be configured to buffer neural network configuration parametersafter reception by the transceiver 1102 and prior to storage by the datastore 1112. As another example, the memory 1106 may be configured tobuffer a state determination value (e.g., “safe driving” or “not safedriving”) after generation by the processor 1108 and before transmissionby the transceiver 1102 to an automobile controller. As another example,the memory 1106 may be configured to store information defining a neuralnetwork structure as generated by the processor 1108 based on the neuralnetwork configuration parameters. The information defining a neuralnetwork structure may be used by the processor 1108 when applying animage received by the transceiver 1102 as input to the neural networkstructure (e.g., neural network structures 600, 650).

The processor 1108 may be configured to process information stored bythe memory 1106, received by the transceiver 1102, and/or stored by thedata store 1112. The processor 1108 may be configured to perform variousdata processing tasks. For example, the processor 1108 may be configuredto generate in memory 1106 a neural network structure based onprocessing the neural network configuration parameters received by thetransceiver 1102. As another example, the processor 1108 may beconfigured to transform an image (e.g., compressing the image,transforming the image to grayscale, enhancing brightness, enhancingcontrast) received by the transceiver 1102. As another example, theprocessor 1108 may be configured to apply the image as input to theneural network structure. As another example, the processor 1108 may beconfigured to determine an output node (e.g., output nodes 675, 676)that has a highest value. As another example, the processor 1108 may beconfigured to determine a category predicted for the image by the neuralnetwork structure based on providing the image as input to the neuralnetwork structure. In some embodiments, the process 1108 may beconfigured to generate a neural network structure and use the neuralnetwork structure on images captured of an interior of an automobile byan image sensor co-located in the automobile with the local computingdevice 1100.

In some embodiments, the processor 1108 may be provided as a singleprocessor as part of the local computing device 1100. In someembodiments, the processor 1108 may be provided as multiple processorsas part of the local computing device 1100. In some embodiments, theprocessor 1108 may be provided as multiple cores of one or moreprocessors as part of the local computing device 1100. In someembodiments, the processor 1108 may be provided as multiple processorsas part of more than one local computing device 1100. Other forms ofprocessor configurations may be used in various embodiments of thepresent disclosure.

The power input 1110 may be configured to provide electric power to thetransceiver 1102, memory 1106, processor 1108, and data store 1112. Insome embodiments, the power input 1110 may include an electricalcoupling to each of the transceiver 1102, memory 1106, processor 1108,and data store 1112 apart from the coupling 1104.

The data store 1112 may be configured to store neural networkconfiguration parameters received by the transceiver 1102 and/or neuralnetwork structure information prepared by the processor 1108. In someembodiments, the data store 1112 may be provided as a single storagedevice as part of the local computing device 1100. In some embodiments,the data store 1112 may be provided as multiple storage devices as partof the local computing device 1100. In some embodiments, the data store1112 may be provided as multiple storage devices as part of more thanone local computing devices 1100. Other forms of storage configurationsmay be used in various embodiments of the present disclosure.

FIG. 12 is a diagram of an automobile controller 1200 according to someembodiments of the present disclosure. The automobile controller 1200may be provided as an automobile controller (e.g., automobile controller728) in some embodiments of the present disclosure. The automobilecontroller 1200 includes a transceiver 1202, a coupling 1204, a memory1206, a processor 1208, a power input 1210, and a data store 1212.

The transceiver 1202, memory 1206, processor 1208, power input 1210, anddata store 1212 may be coupled together by coupling 1204 (e.g., a wiredcommunications bus). In some embodiments, only a subset of transceiver1202, memory 1206, processor 1208, power input 1210, and data store 1212may be coupled together by coupling 1204.

The transceiver 1202 may be configured to communicate with other devicesco-located in an automobile (e.g., local computing device 726). Forexample, the transceiver 1202 may be configured to receive informationpertaining to a current state of a driver (e.g., “safe driver” or “notsafe driving”) of the automobile as determined by a local computingdevice (e.g., local computing device 726, 1100). As another example, thetransceiver 1202 may be configured to receive information pertaining toa current driving mode (e.g., self-driving, manual) of the automobile.As another example, the transceiver 1202 may be configured to receiveinformation pertaining to a current velocity of the automobile. Asanother example, the transceiver 1202 may be configured to receiveinformation pertaining to a current gear of a transmission of theautomobile.

The memory 1206 may be configured to temporarily store informationpertaining to a current state of a driver of the automobile and/orinformation pertaining to a current operation mode (e.g., self-driving,manual) of the automobile. For example, the memory 1206 may beconfigured to buffer driver state information after reception bytransceiver 1202. As another example, the memory 1106 may be configuredto buffer operation mode information that indicates the current mode ofthe automobile (e.g., a self-driving or autopilot mode versus a manualmode).

The processor 1208 may be configured to determine whether to transitionthe automobile between a self-driving or autopilot mode and a manualmode based on the driver state information received by the transceiver1202. For example, the processor 1208 may be determine based on atransition rule stored in data store 1212 that the automobile cannottransition from a self-driving or autopilot mode to a manual mode if thecurrent driver state information indicates that the driver is in an“unsafe driving” state.

The power input 1210 may be configured to provide electric power to thetransceiver 1202, memory 1206, processor 1208, and data store 1212. Insome embodiments, the power input 1210 may include an electricalcoupling to each of the transceiver 1202, memory 1206, processor 1208,and data store 1212 apart from the coupling 1204.

The data store 1212 may be configured to store driving mode transitionrules that specify when the automobile controller 1200 must restricttransition or force transition between a self-driving or autopilot modeand a manual mode (and vice versa) based on a current state of thedriver of the automobile. For example, a driving mode transition rulemay specific that the automobile cannot transition from a self-drivingor autopilot mode to a manual mode if the current driver stateinformation indicates that the driver is in an “unsafe driving” state.As another example, a driving mode transition rule may specific that theautomobile must transition from a manual mode to a self-driving orautopilot mode if the current driver state information indicates thatthe driver is in an “unsafe driving” state and the current velocity ofthe automobile is greater than zero. Other configuration of driving modetransition rules may be used in various embodiments of the presentdisclosure.

FIG. 13 is a system diagram of a system 1300 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 1300 includes an automobile 1310, acomposite sensor/computing device 1316, a remote computing device 1320,a communication link 1322, and an automobile controller 1328.

The automobile 1310 may be an automobile as described previously withrespect to automobiles 110, 300, 400, and/or 710. The automobile 710 maybe operated by a driver seated in driver's seat 1311. The driver mayoperate the automobile 1310 using the steering wheel 1312 and othercontrol devices.

The composite sensor/computing device 1316 may combine the functionalityof a sensor device (e.g., sensor devices 232, 234, 460, 472, 800), alocal transceiver device (e.g., local transceivers 724, 900), and alocal computing device (e.g., local computing devices 726, 1100). Thecomposite sensor/computing device 1316 may be provided at a location1315 built into or adjacent to an infotainment console. The compositesensor/computing device 1316 may be an advantageous configurationcombining the functionality of three different devices. This arrangementmay be preferred in some embodiments due to the simplicity ofinstallation in the automobile and/or the reduced cost in manufacturingand maintaining a single device. In some embodiments, installation ofthe composite sensor/computing device 1316 in a centralized locationsuch as the location 1315 of an infotainment console may be advantageousfor allowing the sensor to sense the position of the driver seated indriver's seat 1311 while also allowing direct connections to otherelectronic components provided behind the dashboard of the automobile.

The remote computing device 1320 may be provided as described previouslywith respect to remote computing devices 130, 720, and/or 1000. Thecommunication link 1322 may be provided as described previously withrespect to communication links 120 and/or 722. The automobile controller1328 may be provided as described previously with respect to automobilecontrollers 728 and/or 1200.

FIG. 14 is a system diagram of a system 1400 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 1400 includes an automobile 1410, acomposite sensor/computing device 1416, a mobile device 1430, a remotecomputing device 1420, a communication link 1422, and an automobilecontroller 1428.

The automobile 1410 may be an automobile as described previously withrespect to automobiles 110, 300, 400, 710, and/or 1310. The automobile1410 may be operated by a driver seated in driver's seat 1411. Thedriver may operate the automobile 1410 using the steering wheel 1412 andother control devices.

The composite sensor/computing device 1416 may combine the functionalityof a sensor device (e.g., sensor devices 232, 234, 460, 472, 800) and alocal computing device (e.g., local computing devices 726, 1100). Thecomposite sensor/computing device 1416 may be provided at a location1415 built into or adjacent to an infotainment console. The compositesensor/computing device 1416 may not contain the functionality of alocal transceiver device (e.g., local transceivers 724, 900). Instead,the composite sensor/computing device 1416 may communicate with mobiledevice 1430 in order to transmit images to and receive neural networkconfiguration parameters from the remote computing device 1420. Thecomposite sensor/computing device 1416 may communicate with the mobiledevice 1430 using short-range wireless communications (e.g., Bluetooth),or using some other communications technology. The mobile device 1430may communicate with the remote computing device 1420 using a long-rangewireless communications link (e.g., CDMA2000, GSM), a short-rangewireless communications link (e.g., IEEE 802.11 Wi-Fi), or using someother communications technology.

The composite sensor/computing device 1416 may be an advantageousconfiguration combining the functionality of two different devices. Thisarrangement may be preferred in some embodiments due to the simplicityof installation in the automobile, the reduced cost in manufacturing andmaintaining a single device, and/or the reduced cost and administrativechallenge of not requiring a dedicated long-range communications link(i.e., using an existing communications link provided by mobile device1430).

The remote computing device 1420 may be provided as described previouslywith respect to remote computing devices 130, 720, 1000, and/or 1310.The communication link 1422 may be provided as described previously withrespect to communication links 120, 722, and/or 1322. The automobilecontroller 1428 may be provided as described previously with respect toautomobile controllers 728, 1200, and/or 1328.

FIG. 15 is a system diagram of a system 1500 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 1500 includes an automobile 1510, asensor 1516, a sensor 1518, a remote computing device 1520, acommunication link 1522, a local transceiver 1524, a local computingdevice 1526, and an automobile controller 1528.

The automobile 1510 may be an automobile as described previously withrespect to automobiles 110, 300, 400, and/or 710. The automobile 1510may be operated by a driver seated in driver's seat 1511. The driver mayoperate the automobile 1510 using the steering wheel 1512 and othercontrol devices. In some embodiments, the automobile 1510 may beconfigured alert the driver of safety hazards or other pertinentconditions. For example, the automobile 1510 may include an indicatorlight and an audible tone to alert the driver that the driver does nothave his safety belt engaged. The automobile 1510 may include a varietyof other indicator devices, including visual indicators, audibleindicators, haptic indicators, electronic transmitters, etc.

The automobile 1510 may further include a rear seat 1513 and a childsafety seat 1514. The child safety seat 1514 may be provided in itsordinary form as a seat in which a baby, toddler, or other child isseated and restrained. The child may be restrained in the child safetyseat 1514 by a five-point harness, three-point harness, or otherharness. The child safety seat 1514 may be attached (e.g., by belts andclips) to the rear seat 1513 to prevent the child safety seat 1514 fromshifting during movement of the automobile 1510 or after collisionbetween the automobile 1510 and some item in its environment.

The sensor 1516 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, 472, 716, and/or 800.The sensor 1516 may be provided at a location 1515, such as in alocation built into or in the vicinity of a rearview mirror.

The sensor 1518 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, 472, 716, and/or 800.The sensor 1518 may be provided at a location 1517, such as affixed tothe interior ceiling the automobile 1510.

In some embodiments, system 1500 may employ a neural network in thefollowing way.

The sensor 1516 may sense the position of the driver seated in driver'sseat 1511. Sensor 1516 may produce sensor data based on this sensing.For example, if the sensor 1516 is a video camera, then the sensor 1516may generate an image of the interior of the automobile 1510, includingthe driver, if present.

The sensor 1518 may sense child safety seat 1514 and any occupantthereof. Sensor 1518 may produce sensor data based on this sensing. Forexample, if the sensor 1518 is a video camera, then the sensor 1518 maygenerate an image of the interior of the automobile 1510, including thechild safety seat 1514 and the child seated therein, if present.

The sensor 1516 may transmit the driver image to the transceiver 1524.This transmission may be performed using a shared communications bus,using a dedicated physical cable, using a short-range wireless link, orin some other way.

The sensor 1518 may transmit the child safety seat image to thetransceiver 1524. This transmission may be performed using a sharedcommunications bus, using a dedicated physical cable, using ashort-range wireless link, or in some other way.

The transceiver 1524 may transmit the driver image and the child safetyseat image to the remote computing device 1520 using the communicationlink 1522. The communication link 1522 may be provided as described withrespect to communication link 722.

The remote computing device 1520 may save the received driver image andchild safety seat image for later processing. In some embodiments, theremote computing device 1520 may transform the driver image and/or thechild safety seat image (e.g., compressing the image, transforming theimage to grayscale, enhancing brightness, enhancing contrast). In someembodiments, the sensor 1516, the sensor 1518, the transceiver 1524, orsome other computing device located in the automobile 1510 (e.g., thelocal computing device 1526) may transform the driver image and/or thechild safety seat image (e.g., compressing the image, transforming theimage to grayscale, enhancing brightness, enhancing contrast) prior totransmitting the driver image and/or the child safety seat image to theremote computing device 1520.

The sensor 1516, the sensor 1518, the transceiver 1524, and the remotecomputing device 1520 may repeat this process of image capture,transmission, and storage numerous times until a large quantity ofimages are stored by the remote computing device 1520. In someembodiments, the large quantity of images may be 100 images. In someembodiments, the large quantity of images may be 1,000 images. In someembodiments, the large quantity of images may be 10,000 images. In someembodiments, the large quantity of images may be 100,000 images. In someembodiments, the sensor 1516, the transceiver 1524, and the remotecomputing device 1520 may perform this process of image capture,transmission, and storage of the driver image independent of the similarprocess for the child safety seat image. Likewise, the sensor 1518, thetransceiver 1524, and the remote computing device 1520 may perform thisprocess of image capture, transmission, and storage of the child safetyseat image independent of the similar process for the driver image.

One or more users may label the driver images based on predefinedcategories (e.g., “present” and “not present”). A user may label adriver image “present” if the image shows a driver present in thedriver's seat 1511. A user may label a driver image “not present” if theimage shows the driver's seat 1511 with no driver seated therein. Theremote computing device 1520 may store these category labels with theimages to which they pertain. In some embodiments, a user located in theautomobile 1510 (e.g., the driver, a passenger) may perform thelabelling prior to transmitting the corresponding driver image to theremote computing device 1520. The one or more users may label the imageswhile the remote computing device 1520 is still waiting to finishstorage of a large quantity of driver images, after completing storageof a large quantity of driver images, or at some other time.

One or more users may label the child safety seat images based onpredefined categories (e.g., “present” and “not present”). A user maylabel a child safety seat image “present” if the image shows a childpresent in the child safety seat 1514. A user may label a child safetyseat image “not present” if the image shows the child safety seat 1514with no child seated therein. The remote computing device 1520 may storethese category labels with the images to which they pertain. In someembodiments, a user located in the automobile 1510 (e.g., the driver, apassenger) may perform the labelling prior to transmitting thecorresponding child safety seat image to the remote computing device1520. The one or more users may label the images while the remotecomputing device 1520 is still waiting to finish storage of a largequantity of child safety seat images, after completing storage of alarge quantity of child safety seat images, or at some other time.

Once a large quantity of driver images and associated category labelsare stored by the remote computing device 1520, the remote computingdevice 1520 may train a neural network structure (e.g., neural networkstructures 600, 650) based on the driver images and correspondinglabels. For example, if the images are images of the interior of thevehicle 1510 including the presence or absence of a driver's body in thedriver's seat 1511, and the category labels are each selected from“present” and “not present,” the remote computing device 1520 may traina neural network structure to accurately categorize the state of thedriver as “present” or “not present” based on the driver's bodyposition. For the remainder of the disclosure of this embodiment, aneural network structure so trained will be referred to as a driverpresence neural network structure.

Once a large quantity of child safety seat images and associatedcategory labels are stored by the remote computing device 1520, theremote computing device 1520 may train a neural network structure (e.g.,neural network structures 600, 650) based on the child safety seatimages and corresponding labels. For example, if the images are imagesof the interior of the vehicle 1510 including the child safety seat andthe presence or absence of a child's body in the child safety seat 1514,and the category labels are each selected from “present” and “notpresent,” the remote computing device 1520 may train a neural networkstructure to accurately categorize the state of the child as “present”or “not present” based on the driver's body presence. For the remainderof the disclosure of this embodiment, a neural network structure sotrained will be referred to as a child presence neural networkstructure.

Once the remote computing device 1520 trains the driver presence neuralnetwork structure, the remote computing device 1520 may transmitparameters defining the driver presence neural network structure (alsoreferred to herein as “neural network configuration parameters”) (e.g.,specification of nodes, specification of combination functions fornodes, specification of activation functions for nodes, specification ofconnections, specification of weights for connections) to the localtransceiver 1524 using the communication link 1522. In some embodiments,the remote computing device 1520 may transmit the parameters definingthe driver presence neural network structure to the local transceiver1524 using a communication link other than communication link 1522.

Once the remote computing device 1520 trains the child presence neuralnetwork structure, the remote computing device 1520 may transmitparameters defining the child presence neural network structure (alsoreferred to herein as “neural network configuration parameters”) (e.g.,specification of nodes, specification of combination functions fornodes, specification of activation functions for nodes, specification ofconnections, specification of weights for connections) to the localtransceiver 1524 using the communication link 1522. In some embodiments,the remote computing device 1520 may transmit the parameters definingthe child presence neural network structure to the local transceiver1524 using a communication link other than communication link 1522.

After receiving the parameters defining the driver presence neuralnetwork structure from the remote computing device 1520, the localtransceiver 1524 may transmit those parameters to the local computingdevice 1526. The local computing device 1526 may be configured toreceive the parameters defining the driver presence neural networkstructure from the local transceiver 1524. The local computing device1526 may also be configured to receive a driver image (as previouslydisclosed) from the sensor 1516. This reception from the localtransceiver 1524 and the sensor 1516 may be performed using any of avariety of communication mediums, such as those described previously forcommunication between the sensor 1516 and the local transceiver 1524.

After receiving the parameters defining the child presence neuralnetwork structure from the remote computing device 1520, the localtransceiver 1524 may transmit those parameters to the local computingdevice 1526. The local computing device 1526 may be configured toreceive the parameters defining the child presence neural networkstructure from the local transceiver 1524. The local computing device1526 may also be configured to receive a child safety seat image (aspreviously disclosed) from the sensor 1518. This reception from thelocal transceiver 1524 and the sensor 1518 may be performed using any ofa variety of communication mediums, such as those described previouslyfor communication between the sensor 1518 and the local transceiver1524.

The local computing device 1526 may be configured to create thefully-trained driver presence neural network structure (e.g., recreatethe structure in local memory) based on the parameters defining thedriver presence neural network structure. Once the local computingdevice 1526 receives the parameters defining the driver presence neuralnetwork structure, the local computing device 1526 may provide a driverimage received from the sensor 1516 as input to the driver presenceneural network structure in real-time. The local computing device 1526may thereby determine a category for the driver image (e.g., “present”or “not present”) which indicates the most likely state of the driver ofthe automobile 1510 when the driver image was captured. If the driverimage is provided to the local computing device 1526 in substantiallyreal-time, then the local computing device 1526 may be able to determinewhether the driver is in a present state or a not present state atsubstantially the present time.

The local computing device 1526 may be configured to create thefully-trained child presence neural network structure (e.g., recreatethe structure in local memory) based on the parameters defining thechild presence neural network structure. Once the local computing device1526 receives the parameters defining the child presence neural networkstructure, the local computing device 1526 may provide a child safetyseat image received from the sensor 1518 as input to the child presenceneural network structure in real-time. The local computing device 1526may thereby determine a category for the child safety seat image (e.g.,“present” or “not present”) which indicates the most likely state of thechild or lack thereof in the child safety seat 1514 when the childsafety seat image was captured. If the child safety seat image isprovided to the local computing device 1526 in substantially real-time,then the local computing device 1526 may be able to determine whetherthe child is in a present state or a not present state at substantiallythe present time.

The local computing device 1526 may transmit the determined state of thedriver (e.g., “present” or “not present”) to the automobile controller1528. This transmission to the automobile controller 1528 may beperformed using any of a variety of communication mediums, such as thosedescribed previously for communication between the sensor 1516 and thelocal transceiver 1524.

The local computing device 1526 may transmit the determined state of thechild (e.g., “present” or “not present”) to the automobile controller1528. This transmission to the automobile controller 1528 may beperformed using any of a variety of communication mediums, such as thosedescribed previously for communication between the sensor 1516 and thelocal transceiver 1524.

The automobile controller 1528 may be configured to control one or moreindicators based on the state of the driver determination received fromthe local computing device 1526 and based on the state of the childdetermination received from the local computing device 1526. Forexample, the automobile controller 1528 may be configured to notactivate an indicator (e.g., warning light, warning audible tone) if thedetermined driver state is “present” while the determined child state isalso “present.” Continuing the example, the automobile controller 1528may be configured to not activate an indicator (e.g., warning light,warning audible tone) if the determined driver state is “present” whilethe determined child state is “not present.” Continuing the example, theautomobile controller 1528 may be configured to not activate anindicator (e.g., warning light, warning audible tone) if the determineddriver state is “not present” while the determined child state is also“not present.” But, continuing the example, the automobile controller1528 may be configured to activate an indicator (e.g., warning light andwarning audible tone) if the determined driver state is “not present”while the determined child state is “present.” This configuration of theautomobile controller 1528 may reflect a design to alert the driver thatmay be at risk of accidently leaving the child in the car, which maypose a mortal risk to the child depending on various factors such as theambient temperature outside the automobile 1510. As such, the automobilecontroller 1528 may be configured to activate an indicator when a highrisk condition is identified through use of the driver presence neuralnetwork structure and the child presence neural network structure.

In some embodiments, the automobile controller 1528 may be configured toactivate an indicator when a more specific condition is met. Forexample, the automobile controller 1528 may be configured to activate anindicator at time t2 when the determined driver state is “not present”while the determined child state is “present,” but only if at time t1, atime immediately preceding time t2, the determined driver state is“present” while the determined child state is also “present.” Thisconfiguration of the automobile controller 1528 may reflect a design toprevent a risk to a child in the child safety seat 1514 while alsoavoiding annoyance to the driver. In particular, this configurationwould avoid activating the indicator during the time between when thedriver first places the child in the child safety seat 1514 and when thedriver himself enters the driver's seat 1511. This configuration maythus more specifically target the high risk condition of a driver havinga child in the child safety seat 1514 and inadvertently leaving thechild there when after arriving at the driver's destination.

In some embodiments, the sensor 1516 may be configured to additionallysense the presence of an adult in a passenger seat (e.g., frontpassenger seat) of the automobile 1510. In such embodiments, the driverimages (which may be considered “adult images”) may be labelled aspreviously described. However, the label “present” may indicate that adriver is present in the automobile 1510, that an adult passenger ispresent in the automobile 1510, or that both are present in theautomobile 1510. Accordingly, the label “not present” may indicate thatno adult is present in the vehicle. In such embodiments, the remotecomputing device 1520 may train and the local computing device 1526 mayuse a neural network structure that determines whether an adult ispresent in the automobile 1510, and not more specifically whether adriver is present in the automobile 1510. Likewise, the automobilecontroller 1528 may control the automobile 1510 based on thisdetermination of whether an adult is present. In some embodiments, thesensor 1518 may likewise detect the presence of an adult in theautomobile 1510 (e.g., in a rear passenger seat). In such embodiments,the sensor data generated by the sensor 1518 may additionally be used todetermine whether an adult is present in the automobile 1510.

Other configurations of the automobile controller 1528 to use areal-time determination based on a neural network structure may be usedin accordance with various embodiments of the present disclosure.

FIG. 16 is a system diagram of a system 1600 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 1600 includes an automobile 1610, asensor 1616, a sensor 1618, a remote computing device 1620, acommunication link 1622, a local transceiver 1624, a local computingdevice 1626, and an automobile controller 1628.

The automobile 1610 may be an automobile as described previously withrespect to automobiles 110, 300, 400, 710, and/or 1510. In someembodiments, the automobile 1610 may be configured alert the driver ofsafety hazards or other pertinent conditions. For example, theautomobile 1610 may include an indicator light and an audible tone toalert the driver of the automobile 1610 that the driver does not havehis safety belt engaged. The automobile 1610 may include a variety ofother indicator devices, including visual indicators, audibleindicators, haptic indicators, electronic transmitters, etc. In someembodiments, the automobile controller 1628 may be configured to controlthe shifting of the automobile 1610 between different transmission gears(e.g., by sending an enabling/disabling signal to an electromechanicalinterlock on a gear shifter).

The automobile 1610 may further include a rear seat 1613 and a childsafety seat 1614. The child safety seat 1614 may be provided asdescribed previously with respect to child safety seat 1514.

The sensor 1616 may be an sensor capable of detecting a state of theautomobile 1610.

For example, the sensor 1616 may be a sensor configured to detect acurrent gear (e.g., “park,” “drive,” “reverse,” “first,” “second”) of atransmission of the automobile 1610. In such an example, the sensor 1616may be provided as an optical sensor attached to a gear shifter in theautomobile 1610 and configured to detect the current gear based on theposition of the gear shifter. In other cases, the sensor 1616 may be anelectro-mechanical device attached to the transmission and configured todetect the current gear based on the position of one or more gear shiftrods of the transmission. In other cases, the sensor 1616 may beprovided as an electronic module connected to a communications bus ofthe automobile 1610 and configured to detect the current gear based on asignal passed on the communications bus.

As another example, the sensor 1616 may be a sensor configured to detecta current velocity of the automobile 1610. For example, the sensor 1616may be provided as an optical sensor attached to a tire of theautomobile 1610 and configured to detect the current velocity based onthe rate of revolution of the tire. In other cases, the sensor 1616 maybe provided as a Global Positioning System receiver and configured todetect the current velocity based on the rate of change of the positionof the automobile 1610. In other cases, the sensor 1616 may be providedas an electronic module connected to a communications bus of theautomobile 1610 and configured to detect the current velocity based on asignal passed on the communications bus.

As another example, the sensor 1616 may be a sensor configured to detecta current motion state of the automobile 1610, such as a “drivingforward” state, a “driving in reverse” state, and a “parked” state.

The sensor 1616 may be provided in other forms to detect a state of theautomobile 1610 consistent with various embodiments of the presentdisclosure.

The sensor 1618 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, 472, 716, 800, and/or1518. The sensor 1618 may be provided at a location 1617, such asaffixed to the interior ceiling the automobile 1610.

In some embodiments, system 1600 may employ a neural network in thefollowing way.

The sensor 1618 may sense the position of the child seated in the childsafety seat 1614. Sensor 1618 may produce sensor data based on thissensing. For example, if sensor 1618 is a video camera, then the sensor1618 may generate an image of the interior of the automobile 1610,including the child safety seat 1614, any visible portions of therestrains of the child safety seat 1614 (e.g., belts, buckles, clips),and position of the child in the safety seat 1614, if present. The imagemay include the position of the child's legs, arms, torso, head, etc.and their position with respect to the child safety seat 1614 and therestraints of the child safety seat 1614.

The sensor 1618 may transmit the image to the transceiver 1624. Thistransmission may be performed using a shared communications bus, using adedicated physical cable, using a short-range wireless link, or in someother way.

The transceiver 1624 may transmit the image to the remote computingdevice 1620 using the communication link 1622. The communication link1622 may be provided as described with respect to communication links722 and/or 1522.

The remote computing device 1620 may save the received image for laterprocessing. In some embodiments, the remote computing device 1620 maytransform the image (e.g., compressing the image, transforming the imageto grayscale, enhancing brightness, enhancing contrast). In someembodiments, the sensor 1618, the transceiver 1624, or some othercomputing device located in the automobile 1610 (e.g., the localcomputing device 1626) may transform the image (e.g., compressing theimage, transforming the image to grayscale, enhancing brightness,enhancing contrast) prior to transmitting the image to the remotecomputing device 1620.

The sensor 1618, the transceiver 1624, and the remote computing device1620 may repeat this process of image capture, transmission, and storagenumerous times until a large quantity of images are stored by the remotecomputing device 1620. In some embodiments, the large quantity of imagesmay be 100 images. In some embodiments, the large quantity of images maybe 1,000 images. In some embodiments, the large quantity of images maybe 10,000 images. In some embodiments, the large quantity of images maybe 100,000 images.

One or more users may label the images based on predefined categories(e.g., “restrained” and “not restrained”). A user may label an image“restrained” if the image shows a child properly restrained in the childsafety seat 1614 (e.g., should, waist, and groin belts properly located,five-point buckle engaged, chest clip engaged). A user may label animage “not restrained” if the image shows the child improperlyrestrained in the child safety seat 1614 (e.g., five-point buckle notengaged, chest clip not engaged, shoulder belts behind the shoulders,child completely unrestrained and sitting on the front edge of the childsafety seat 1614, child completely unrestrained and standing in thechild safety seat 1614). The remote computing device 1620 may storethese category labels with the images to which they pertain. In someembodiments, a user located in the automobile 1610 (e.g., the driver, apassenger) may perform the labelling prior to transmitting thecorresponding image to the remote computing device 1620. The one or moreusers may label the images while the remote computing device 1620 isstill waiting to finish storage of a large quantity of images, aftercompleting storage of a large quantity of images, or at some other time.

Once a large quantity of images and associated category labels arestored by the remote computing device 1620, the remote computing device1620 may train a neural network structure (e.g., neural networkstructures 600, 650) based on the images and corresponding labels. Forexample, if the images are images of the interior of the vehicle 1610including the child safety seat 1614, any visible portions of therestrains of the child safety seat 1614 (e.g., belts, buckles, clips),and the position of the child's body (e.g., legs, arms, torso, head,etc.), and the category labels are each selected from “restrained” and“not restrained,” then the remote computing device 1620 may train aneural network structure to accurately categorize the state of the childas “restrained” or “not restrained” based on the child's body position.

Once the remote computing device 1620 trains the neural networkstructure, the remote computing device 1620 may transmit parametersdefining the neural network structure (also referred to herein as“neural network configuration parameters”) (e.g., specification ofnodes, specification of combination functions for nodes, specificationof activation functions for nodes, specification of connections,specification of weights for connections) to the local transceiver 1624using the communication link 1622. In some embodiments, the remotecomputing device 1620 may transmit the parameters defining the neuralnetwork structure to the local transceiver 1624 using a communicationlink other than communication link 1622.

After receiving the parameters defining the neural network structurefrom the remote computing device 1620, the local transceiver 1624 maytransmit those parameters to the local computing device 1626. The localcomputing device 1626 may be configured to receive the parametersdefining the neural network structure from the local transceiver 1624.The local computing device 1626 may also be configured to receive achild safety seat image (as previously disclosed) from the sensor 1618.This reception from the local transceiver 1624 and the sensor 1618 maybe performed using any of a variety of communication mediums, such asthose described previously for communication between the sensor 1618 andthe local transceiver 1624.

The local computing device 1626 may be configured to create thefully-trained neural network structure (e.g., recreate the structure inlocal memory) based on the parameters defining the neural networkstructure. Once the local computing device 1626 receives the parametersdefining the neural network structure, the local computing device 1626may provide a child safety seat image received from the sensor 1618 asinput to the neural network structure in real-time. The local computingdevice 1626 may thereby determine a category for the child safety seatimage (e.g., “restrained” or “not restrained”) which indicates the mostlikely state of the child in the child safety seat 1614 when the childsafety seat image was captured. If the child safety seat image isprovided to the local computing device 1626 in substantially real-time,then the local computing device 1626 may be able to determine whetherthe child is in a restrained state or a not restrained state atsubstantially the present time.

The local computing device 1626 may transmit the determined state of thechild (e.g., “restrained” or “not restrained”) to the automobilecontroller 1628. This transmission to the automobile controller 1628 maybe performed using any of a variety of communication mediums, such asthose described previously for communication between the sensor 1618 andthe local transceiver 1624.

The automobile controller 1628 may be configured to receive the state ofthe child determination from the local computing device 1626, and astate of the automobile 1610 determination (e.g., a current gear, acurrent velocity) from the sensor 1616. This reception from the localcomputing device 1626 and the sensor 1616 may be performed using any ofa variety of communication mediums, such as those described previouslyfor communication between the sensor 1618 and the local transceiver1624.

In some embodiments, the automobile controller 1628 may be configured tocontrol the operation of the automobile 1610 based on the state of thechild determination received from the local computing device 1526 andbased on the state of the automobile 1610 determination received fromthe sensor 1616. For example, the automobile controller 1628 may beconfigured to transmit a “disable” command to an electromechanicalinterlock on a gear shifter of the automobile 1610 if the determinedchild state is “not restrained” and the vehicle state is “in park gear.”Continuing the example, the automobile controller 1628 may be configuredto transmit an “enable” command to an electromechanical interlock on agear shifter of the automobile 1610 if the determined child state is“not restrained” and the vehicle state is “in drive gear.” Continuingthe example, the automobile controller 1628 may be configured totransmit an “enable” command to an electromechanical interlock on a gearshifter of the automobile 1610 if the determined child state is“restrained” and the vehicle state is “in park gear.” Continuing theexample, the automobile controller 1628 may be configured to transmit an“enable” command to an electromechanical interlock on a gear shifter ofthe automobile 1610 if the determined child state is “restrained” andthe vehicle state is “in drive gear.” This configuration of theautomobile controller 1628 may reflect a design to prevent the driverfrom operating the automobile 1610 while having forgotten to restrainthe child in the child safety seat 1614, which may pose a mortal risk tothe child in the case of a collision of the automobile 1610 with anelement in its environment. As such, the automobile controller 1628 maybe configured to prevent a high risk condition as identified through useof the neural network structure.

In some embodiments, the automobile controller 1628 may be configured tocontrol one or more indicators based on the state of the childdetermination received from the local computing device 1526 and based onthe state of the automobile 1610 determination received from the sensor1616. For example, the automobile controller 1628 may be configured tonot activate an indicator (e.g., warning light, warning audible tone) ifthe determined child state is “restrained” and the vehicle state is “indrive gear.” Continuing the example, the automobile controller 1628 maybe configured to not activate an indicator (e.g., warning light, warningaudible tone) if the determined child state is “restrained” and thevehicle state is “in park gear.” Continuing the example, the automobilecontroller 1628 may be configured to activate an indicator (e.g.,warning light, warning audible tone) if the determined child state is“not restrained” and the vehicle state is “in drive gear.” Continuingthe example, the automobile controller 1628 may be configured to notactivate an indicator (e.g., warning light, warning audible tone) if thedetermined child state is “not restrained” and the vehicle state is “inpark gear.” This configuration of the automobile controller 1628 mayreflect a design to alert the driver that may be at risk of operatingthe automobile 1610 while having forgotten to restrain the child in thechild safety seat 1614, which may pose a mortal risk to the child in thecase of a collision of the automobile 1610 with an element in itsenvironment. As such, the automobile controller 1628 may be configuredto activate an indicator when a high risk condition is identifiedthrough use of the neural network structure.

As another example, the automobile controller 1628 may be configured tonot activate an indicator (e.g., warning light, warning audible tone) ifthe determined child state is “restrained” and the vehicle state is avelocity of zero. Continuing the example, the automobile controller 1628may be configured to not activate an indicator (e.g., warning light,warning audible tone) if the determined child state is “restrained” andthe vehicle state is a velocity other than zero. Continuing the example,the automobile controller 1628 may be configured to activate anindicator (e.g., warning light, warning audible tone) if the determinedchild state is “not restrained” and the vehicle state is a velocityother than zero. Continuing the example, the automobile controller 1628may be configured to not activate an indicator (e.g., warning light,warning audible tone) if the determined child state is “not restrained”and the vehicle state is “a velocity of zero. This configuration of theautomobile controller 1628 may reflect a design to alert the driver thatmay be at risk of operating the automobile 1610 while having forgottento restrain the child in the child safety seat 1614, which may pose amortal risk to the child in the case of a collision of the automobile1610 with an element in its environment. As such, the automobilecontroller 1628 may be configured to activate an indicator when a highrisk condition is identified through use of the neural networkstructure.

Other configurations of the automobile controller 1628 to use areal-time determination based on a neural network structure may be usedin accordance with various embodiments of the present disclosure.

In some embodiments of the present disclosure, the remote computingdevice 1520 may be configured to train a single neural network structurefor categorizing both a driver state and a child state. For example, ifthe driver image (as captured by the sensor 1516 and potentiallycompressed) is a 100×100 pixel image, then the grayscale pixel valuesfor the driver image may be serially provided to the first 10,000 inputnodes of the neural network structure (e.g., as described with respectto neural network structure 650). If the child safety seat image (ascaptured by the sensor 1518 and potentially compressed) is a 100×100pixel image, then the grayscale pixel values for the child safety seatimage may be serially provided to the next 10,000 input nodes of theneural network structure. The output nodes of the neural networkstructure may then correspond to the following states: “driver and childboth present,” “driver and child both not present,” “driver present butchild not present,” and “driver not present but child present.” In suchembodiments, the remote computing device 1520 may train a single neuralnetwork structure according to this configuration, and the localcomputing device 1526 may use a single neural network structure. Theautomobile controller may then activate an indicator for only a subsetof these driver/child states (e.g., only for “driver not present butchild present.”). Other configurations of providing sensor images asinput to a neural network may be used in various embodiments of thepresent disclosure.

In some embodiments, the system 1500 may use only a single sensor tosense the presence or absence of both the driver and the child. Forexample, a sensor affixed to the interior ceiling of the automobile atposition 358 (see FIG. 3) may be capable of sensing both the presence orabsence of the driver in the driver's seat 1511 and the presence orabsence of the child in the child safety seat 1514 with a single image.In such embodiments, the remote computing device 1520 may train only asingle neural network structure, and the local computing device 1526 mayuse only a single neural network structure.

FIG. 17 is a system diagram of a system 1700 employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure. The system 1700 includes an automobile 1710, asensor 1716, a remote computing device 1720, a communication link 1722,a local transceiver 1724, a local computing device 1726, and anautomobile controller 1728.

The automobile 1710 may be an automobile as described previously withrespect to automobiles 110, 300, 400, 710, 1510, and/or 1610. Theautomobile 1710 may be operated by a driver seated in driver's seat1711. The driver may operate the automobile 1710 using the steeringwheel 1712 and other control devices. In some embodiments, theautomobile 1710 may be configured alert the driver of safety hazards orother pertinent conditions. For example, the automobile 1710 may includean indicator light and an audible tone to alert the driver that thedriver does not have his safety belt engaged. The automobile 1710 mayinclude a variety of other indicator devices, including visualindicators, audible indicators, haptic indicators, electronictransmitters, etc.

The sensor 1716 may be provided in whole or in part as describedpreviously with respect to sensors 232, 234, 460, 472, 716, 1516, and/or1616. The sensor 1716 may be provided at a location 1715, such as in alocation built into or in the vicinity of dome light on the interiorceiling of the automobile 1710.

In some embodiments, system 1700 may employ a neural network in thefollowing way.

Sensor 1716 may sense the position of the items present in the interiorof the automobile 1710. Sensor 1716 may produce sensor data based onthis sensing. For example, if sensor 1716 is a video camera, the sensor1716 may generate an image of the interior of the automobile 1710,including the seats, steering wheel, center console, and anynon-permanent items present in the automobile 1710. For instance, theimage generated by the sensor 1716 may show the driver's seat 1711 aswell as a wallet 1746 resting on the driver's seat 1711. The image mayalso show a can of soda 1744 resting on the center console, and anumbrella 1742 resting on the floor. In this way, the sensor 1716 maysense both the standard configuration of the interior of the automobile1710 as well as items abandoned in the interior of the automobile 1710by some previous driver.

The sensor 1716 may transmit a an image of the interior of theautomobile 1710 to transceiver 1724. This transmission may be performedusing a shared communications bus, using a dedicated physical cable,using a short-range wireless link, or in some other way.

The transceiver 1724 may transmit the image to the remote computingdevice 1720 using the communication link 1722. The communication link1722 may be provided described herein with respect to communicationlinks 722, 1522, and/or 1622.

The remote computing device 1720 may save the received image for laterprocessing. In some embodiments, the remote computing device 1720 maytransform the image (e.g., compressing the image, transforming the imageto grayscale, enhancing brightness, enhancing contrast). In someembodiments, the sensor 1716, the transceiver 1724, or some othercomputing device located in the automobile 1710 (e.g., the localcomputing device 1726) may transform the image (e.g., compressing theimage, transforming the image to grayscale, enhancing brightness,enhancing contrast) prior to transmitting the image to the remotecomputing device 1720.

The sensor 1716, the transceiver 1724, and the remote computing device1720 may repeat this process of image capture, transmission, and storagenumerous times until a large quantity of images are stored by the remotecomputing device 1720. In some embodiments, the large quantity of imagesmay be 100 images. In some embodiments, the large quantity of images maybe 1,000 images. In some embodiments, the large quantity of images maybe 10,000 images. In some embodiments, the large quantity of images maybe 100,000 images.

One or more users may label the images based on predefined categories(e.g., “no abandoned items”; “abandoned items”; “wallet in seat”;“umbrella on floor”; “beverage in center console”). The remote computingdevice 1720 may store these category labels with the images to whichthey pertain. In some embodiments, a user located in the automobile 1710(e.g., the driver, a passenger) may perform the labelling prior totransmitting the corresponding image to the remote computing device1720. The one or more users may label the images while the remotecomputing device 1720 is still waiting to finish storage a largequantity of images, after completing storage of a large quantity ofimages, or at some other time.

Once a large quantity of images and associated category labels arestored by the remote computing device 1720, the remote computing device1720 may train a neural network structure (e.g., neural networkstructures 600, 650) based on the images and corresponding labels. Forexample, if the images are images of the interior of the vehicle 710 andthe category labels are each selected from “abandoned items” and “noabandoned items,” the remote computing device 1720 may train a neuralnetwork structure to accurately categorize the state of the automobile1710 as having “abandoned items” or having “no abandoned items” based onthe images of the interior of the automobile 1710.

Once the remote computing device 1720 trains the neural networkstructure, the remote computing device 1720 may transmit parametersdefining the neural network structure (also referred to herein as“neural network configuration parameters”) (e.g., specification ofnodes, specification of combination functions for nodes, specificationof activation functions for nodes, specification of connections,specification of weights for connections) to the local transceiver 1724using the communication link 1722. In some embodiments, the remotecomputing device 1720 may transmit the parameters defining the neuralnetwork structure to the local transceiver 1724 using a communicationlink other than communication link 1722.

The local transceiver 1724 may transmit the parameters defining theneural network structure to the local computing device 1726. The localcomputing device 1726 may be configured to receive the parametersdefining the neural network structure from the local transceiver 1724.The local computing device 1726 may also be configured to receive animage of the interior of the automobile 1710 from the sensor 1716. Thisreception from the local transceiver 1724 and the sensor 1716 may beperformed using any of a variety of communication mediums, such as thosedescribed previously for communication between the sensor 1716 and thelocal transceiver 1724.

The local computing device 1726 may be configured to create thefully-trained neural network structure (e.g., recreate the structure inlocal memory) based on the parameters defining the neural networkstructure. Once the local computing device 1726 receives the parametersdefining the neural network structure, the local computing device 1726may provide an image of the interior of the automobile 1710 as input tothe neural network structure in real-time. The local computing device1726 may thereby determine a category for the image of the interior ofthe automobile 1710 (e.g., “abandoned items” or “no abandoned items”)which indicates the most likely state of the automobile 1710 when theimage was captured. If the images of the interior of the automobile 1710are provided to the local computing device 1726 in substantiallyreal-time, then the local computing device 1726 may be able to determinewhether the automobile 1710 contains abandoned items or does not containabandoned items at substantially the present time.

The local computing device 1726 may transmit the determined state of theautomobile (e.g., “abandoned items” or “no abandoned items”) to theautomobile controller 1728. This transmission to the automobilecontroller 1728 may be performed using any of a variety of communicationmediums, such as those described previously for communication betweenthe sensor 1716 and the local transceiver 1724.

The automobile controller 1728 may be configured to control one or moreindicators based on the state of the automobile determination receivedfrom the local computing device 1726. For example, the automobilecontroller 1728 may be configured to not activate an indicator (e.g.,warning light, warning audible tone) if the determined automobile stateis “no abandoned items.” Continuing the example, the automobilecontroller 1728 may be configured to not activate an indicator (e.g.,warning light, warning audible tone) if the determined automobile stateis “abandoned items.”

In some embodiments, the automobile controller 1728 may be configured toactivate an indicator only if the determined automobile state is“abandoned items” and if a user input is received. For instance, acommand may be received from a technician on a user interface (e.g., aninfotainment display, a smartphone connected via short-range wirelesscommunications) indicating that the automobile controller 1728 shoulddisplay an indicator of the state of the automobile 1728. In such acase, the automobile controller 1728 may only activate the indicatorafter receiving the user input and if the determined automobile state is“abandoned items.”

In some embodiments, the user input may be received only after a driverreturns the automobile 1710 to a car rental service or a rideshareparking location. In some embodiments, the activation of the indicatormay be instigated not by a user input but by an automated input (e.g.,electronic signal received by a transponder at a rental service garage,electronic signal received by a transponder at a rideshare parkinglocation, electronic signal generated on a periodic basis by a deviceinside the automobile 1710).

In some embodiments, the user input and/or automated input may bereceived by the local computing device 1726. In such embodiments, theuser input and/or automated input may control whether the localcomputing device 1726 provides an image from the sensor 1716 as an inputto the neural network structure.

In some embodiments of the present disclosure, different categories thanthose just described for system 1700 may be used to categorize a stateof the automobile. For example, the system 1700 may include a neuralnetwork structure trained to determine an automobile state as one of thefollowing: “no abandoned items,” “wallet in seat,” “beverage incupholder,” “umbrella on floor,” and “bag in backseat foot well.” Insuch embodiments, the images used by the remote computing device 1720 totrain the neural network structure may be labeled with these categoriesso that the trained neural network structure may accurately determinethe automobile's state from these categories. In such embodiments, theautomobile controller 1728 may be configured to activate an indicatorfor only some of the automobile states (e.g., for “wallet in seat,”“beverage in cupholder,” “umbrella on floor,” and “bag in backseat footwell” but not for “no abandoned items”).

Other categories of automobile state are possible in various embodimentsof the present disclosure. For example, the system 1700 may include aneural network structure trained to determine an automobile state as oneof the following: “normal state,” “driver's seat reclined,” “frontpassenger seat reclined,” “steering wheel lowered below normal,”“steering wheel elevated above normal,” “driver's side footwell soiled,”“passenger's side footwell soiled,” “trash,” “no trash,” “acceptable”(e.g., to represent that the automobile is clean, empty, free ofabandoned items, etc.; to represent that the automobile is in anacceptable condition to issue to a new driver), and “not acceptable”(e.g., to represent that the automobile is dirty, occupied, containsabandoned items, etc.; to represent that the automobile is not in anacceptable condition to issue to a new driver). Other categories ofautomobile state may be used in accordance with various embodiments ofthe present disclosure.

In some embodiments of the present disclosure, a system may be providedthat combines the sensing and control techniques of more than one of theembodiments described herein. For example, a system may be provided withsensors 1516 and 1518 (see FIG. 15). The remote computing device (e.g.,remote computing device 720, 1520, or 1620) may then use the images fromthe sensor 1516 to train a driver state neural network structure (e.g.,as described with respect to the remote computing device 720). Theremote computing device (e.g., remote computing device 720, 1520, or1620) may then use the images from the sensor 1516 to train a driverpresence neural network structure (e.g., as described with respect tothe remote computing device 1520). The remote computing device (e.g.,remote computing device 720, 1520, or 1620) may then use the images fromthe sensor 1518 to train a child presence neural network structure(e.g., as described with respect to the remote computing device 1520).The remote computing device (e.g., remote computing device 720, 1520, or1620) may then use the images from the sensor 1518 to train a childstate neural network structure (e.g., as described with respect to theremote computing device 1620). The remote computing device may thenprovide each of these neural network structures to the local computingdevice, which may use them in real-time to make each of thedeterminations as descried with respect to local computing devices 726,1526, and 1626. The automobile controller may then be able to performeach of the control techniques described with respect to automobilecontrollers 728, 1528, and 1628 (e.g., driving mode control, indicatorcontrol, gearshift control). Such a configuration may be advantageous asleveraging a small number of sensors and embedded computing resources inthe automobile in order to provide multiple applications of neuralnetwork in the automobile environment.

FIG. 18 is a flowchart of a process for employing a neural network in aresource-constrained environment according to some embodiments of thepresent disclosure.

At block 1802, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 1804, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 1806, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 1806.

At block 1808, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 808 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 1804 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 1808 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 1804(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 1810, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 1810 is captured by the same one or more sensors used to capturesensor data at block 1802 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 1810 iscaptured by a different one or more sensors used to capture sensor dataat block 1802 (e.g., a different video camera, a different infraredcamera).

At block 1812, the sensor data captured at block 1810 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 1812 may include providing the sensor datacaptured at block 1810 as input to the neural network structure.

FIG. 19 is a flowchart of a process 1900 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure.

At block 1902, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 1904, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 1906, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 1906.

At block 1908, the neural network structure is optimized. In someembodiments, block 1908 may include reducing a number of nodes in theneural network structure. In some embodiments, block 1908 may includereducing a number of connections in the neural network structure. Insome embodiments, block 1908 may include reducing a file size of a filestoring parameters defining the neural network structure.

At block 1910, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 1910 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 1904 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 1910 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 1904(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 1912, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 1912 is captured by the same one or more sensors used to capturesensor data at block 1902 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 1912 iscaptured by a different one or more sensors used to capture sensor dataat block 1902 (e.g., a different video camera, a different infraredcamera).

At block 1914, the sensor data captured at block 1912 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 1914 may include providing the sensor datacaptured at block 1912 as input to the neural network structure.

At block 1916, a determination is made as to whether the neural networkstructure is to be updated. In some embodiments, block 1916 may includedetermining if the accuracy of the application of the neural networkstructure at block 1914 has dropped below a threshold. In someembodiments, block 1916 may include determining if the application ofthe neural network structure at block 1914 failed to sufficientlydistinguish between two categories of input sensor data (e.g. a “safedriving” image is indistinguishable from an “unsafe driving” image). Insome embodiments, block 1916 may include determining that a specifiedperiod of time has elapsed since the neural network structure wastransferred to the resource constrained environment as part of block1910.

If the determination at block 1916 results in a determination that theneural network structure is not to be updated, then process 1900continues at blocks 1912 and 1914 with the further capture of sensordata and further application of the neural network structure.

If the determination at block 1916 results in a determination that theneural network structure is to be updated, then process 1900 continuesat block 1902 with the capture of additional sensor data. In someembodiments, determination at block 1916 that the neural networkstructure is to be updated may result in a new neural network structurebeing trained (e.g., at a new iteration of block 1906) based on newsensor data (e.g., sensor data captured at a new iteration of block1902). In some embodiments, determination at block 1916 that the neuralnetwork structure is to be updated may result in an update beingdetermined for the same neural network structure (e.g., a modificationto the weights for one or more connections for the neural networkstructure trained in a previous iteration of the block 1906).

FIG. 20 is a flowchart of a process 2000 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure.

At block 2002, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 2004, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2006, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2006.

At block 2008, the neural network structure is optimized. In someembodiments, block 2008 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2008 may includereducing a number of connections in the neural network structure.

At block 2010, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2010 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2004 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2010 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2004(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2012, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2012 is captured by the same one or more sensors used to capturesensor data at block 2002 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2012 iscaptured by a different one or more sensors used to capture sensor dataat block 2002 (e.g., a different video camera, a different infraredcamera).

At block 2014, the sensor data captured at block 2012 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2014 may include providing the sensor datacaptured at block 2012 as input to the neural network structure.

At block 2016, a determination is made as to whether the neural networkstructure is to be updated. In some embodiments, block 2016 may includedetermining if the accuracy of the application of the neural networkstructure at block 2014 has dropped below a threshold. In someembodiments, block 2016 may include determining if the application ofthe neural network structure at block 2014 failed to sufficientlydistinguish between two categories of input sensor data (e.g. a “safedriving” image is indistinguishable from an “unsafe driving” image). Insome embodiments, block 2016 may include determining that a specifiedperiod of time has elapsed since the neural network structure wastransferred to the resource constrained environment as part of block2010.

If the determination at block 2016 results in a determination that theneural network structure is not to be updated, then process 2000continues at blocks 2012 and 2014 with the further capture of sensordata and further application of the neural network structure.

If the determination at block 2016 results in a determination that theneural network structure is to be updated, then process 2000 continuesat block 2018.

At block 2018, a determination is made as to whether a communicationlink is available. In some embodiments, the block 2018 may includedetermining whether an intermittent wireless communication link ispresently available between the resource-constrained environment and thenon-resource-constrained environment.

If the determination at block 2018 results in a determination that acommunication link is not available, then the process 2000 continues atthe block 2020. At block 2020, waiting is performed for the availabilityof a communications link. In some embodiments, waiting is performed fora predefined period of time before again proceeding to block 2018. Insome embodiments, waiting is performed indefinitely until an interruptsignal is received, at which point the process 2000 proceeds again toblock 2018.

If the determination at block 2018 results in a determination that acommunication link is available, then the process 2000 continues atblock 2002 with the capture of additional sensor data. In someembodiments, determination at block 2018 that a communications link isavailable may result in a new neural network structure being trained(e.g., at a new iteration of block 2006) based on new sensor data (e.g.,sensor data captured at a new iteration of block 2002). In someembodiments, determination at block 2016 that the neural networkstructure is to be updated may result in an update being determined forthe same neural network structure (e.g., a modification to the weightsfor one or more connections for the neural network structure trained ina previous iteration of the block 2008).

Process 2000 may be modified in accordance with various embodiments ofthe present disclosure. For example, in some embodiments, thedetermination of whether a communications link is available (e.g., block2018) may be performed after block 2002 and before block 2004, and/orafter block 2008 and before block 2010.

In some embodiments, a determination at block 2018 that a communicationlink is available may result skipping execution of the block 2002 andcontinuing instead to execution of the block 2004. In such embodiments,the execution of the block 2004 may include transferring outlier sensordata, previously captured and stored in the resource-constrainedenvironment, to the non-resource-constrained environment. In suchembodiments, the outlier sensor data may include sensor data that couldnot be categorized to a high degree of confidence during execution ofthe block 2014. In such embodiments, the subsequent execution of theblock 2006 may include updating the neural network structure using theoutlier sensor data in addition to other sensor data (e.g., the sensordata previously used to train the neural network structure during the aprevious execution of the block 2006).

FIG. 21 is a flowchart of a process 2100 for employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure.

At block 2102, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 2104, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2106, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2106.

At block 2108, the neural network structure is optimized. In someembodiments, block 2108 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2108 may includereducing a number of connections in the neural network structure.

At block 2110, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2110 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2104 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2110 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2104(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2112, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2112 is captured by the same one or more sensors used to capturesensor data at block 2102 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2112 iscaptured by a different one or more sensors used to capture sensor dataat block 2102 (e.g., a different video camera, a different infraredcamera).

At block 2114, the sensor data captured at block 2112 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2114 may include providing the sensor datacaptured at block 2112 as input to the neural network structure.

At block 2116, a determination is made as to whether a driver of theautomobile is in an unsafe driving state. The determination at block2116 may be performed based on an output of the application of theneural network structure to the captured sensor data at the block 2114.

If the determination at block 2116 results in a determination that thedriver is in an unsafe driving state, then the process continues atblock 2118.

At the block 2118, a determination is made as to whether the automobileis presently in a self-driving mode. If the determination at block 2118results in a determination that automobile is presently in aself-driving mode, then the process continues at block 2124. At block2124, the automobile is restricted from transitioning from theself-driving mode to a manual driving mode. If the determination atblock 2118 results in a determination that automobile is not presentlyin a self-driving mode, then the process continues at block 2122. Atblock 2122, the automobile is transitioned to a self-driving mode (e.g.,from a manual driving mode).

If the determination at block 2116 results in a determination that thedriver is not in an unsafe driving state, then the process continues atblock 2120.

At the block 2120, the automobile maintains its current driving mode.

FIG. 22 is a flowchart of a process 2200 for employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure.

At block 2202, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 2204, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2206, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2206.

At block 2208, the neural network structure is optimized. In someembodiments, block 2208 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2208 may includereducing a number of connections in the neural network structure.

At block 2210, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2210 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2204 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2210 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2204(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2212, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2212 is captured by the same one or more sensors used to capturesensor data at block 2202 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2212 iscaptured by a different one or more sensors used to capture sensor dataat block 2202 (e.g., a different video camera, a different infraredcamera).

At block 2214, the sensor data captured at block 2212 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2214 may include providing the sensor datacaptured at block 2212 as input to the neural network structure.

At block 2216, a determination is made as to whether a child is present.In some embodiments, block 2216 may include determining whether a childis present in a child safety seat of the automobile. The determinationat block 2216 may be performed based on an output of the application ofthe neural network structure to the captured sensor data at the block2214.

If the determination at block 2216 results in a determination that achild is not present, then the process continues at block 2222. At block2222, an indicator is not activated (e.g., audible warning tone notemitted).

If the determination at block 2216 results in a determination that achild is present, then the process continues at block 2218.

At the block 2218, a determination is made as to whether adult ispresent. In some embodiments, block 2218 may include determining whetheradult is present in a driver's seat of the automobile or a passengerseat of the automobile. The determination at block 2218 may be performedbased on an output of the application of the neural network structure tothe captured sensor data at the block 2214.

If the determination at block 2218 results in a determination that anadult is not present, then the process continues at block 2220. At block2220, an indicator is activated (e.g., audible warning tone emitted).

If the determination at block 2218 results in a determination that anadult is present, then the process continues at block 2222. At block2222, an indicator is not activated (e.g., audible warning tone notemitted).

FIG. 23 is a flowchart of a process 2300 for employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure.

At block 2302, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 2304, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2306, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2306.

At block 2308, the neural network structure is optimized. In someembodiments, block 2308 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2308 may includereducing a number of connections in the neural network structure.

At block 2310, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2310 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2304 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2310 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2304(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2312, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2312 is captured by the same one or more sensors used to capturesensor data at block 2302 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2312 iscaptured by a different one or more sensors used to capture sensor dataat block 2302 (e.g., a different video camera, a different infraredcamera).

At block 2314, the sensor data captured at block 2312 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2314 may include providing the sensor datacaptured at block 2312 as input to the neural network structure.

At block 2316, a determination is made as to whether a child isrestrained. In some embodiments, block 2316 may include determiningwhether a child is restrained in a child safety seat of the automobile.The determination at block 2316 may be performed based on an output ofthe application of the neural network structure to the captured sensordata at the block 2314.

If the determination at block 2316 results in a determination that achild is restrained, then the process continues at block 2322. At block2322, an indicator is not activated (e.g., audible warning tone notemitted).

If the determination at block 2316 results in a determination that achild is not restrained, then the process continues at block 2318.

At the block 2318, a determination is made as to whether the automobileis in motion. In some embodiments, block 2318 may include determiningwhether the automobile has a velocity other than zero. In someembodiments block 2318 may include determining whether a transmission ofthe automobile is in a gear other than the park gear.

If the determination at block 2318 results in a determination that theautomobile is in motion, then the process continues at block 2320. Atblock 2320, an indicator is activated (e.g., audible warning toneemitted). In some embodiments, the block 2320 may further includeautomatically applying a brake in the automobile.

If the determination at block 2318 results in a determination that theautomobile is not in motion, then the process continues at block 2322.At block 2322, an indicator is not activated (e.g., audible warning tonenot emitted).

FIG. 24 is a flowchart of a process 2400 for employing a neural networkin an automotive environment according to some embodiments of thepresent disclosure.

At block 2402, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 2404, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2406, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2406.

At block 2408, the neural network structure is optimized. In someembodiments, block 2408 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2408 may includereducing a number of connections in the neural network structure.

At block 2410, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2410 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2404 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2410 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2404(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2412, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2412 is captured by the same one or more sensors used to capturesensor data at block 2402 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2412 iscaptured by a different one or more sensors used to capture sensor dataat block 2402 (e.g., a different video camera, a different infraredcamera).

At block 2414, the sensor data captured at block 2412 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2414 may include providing the sensor datacaptured at block 2412 as input to the neural network structure.

At block 2316, a determination is made as to whether an abandoned itemis present in the automobile. The determination at block 2416 may beperformed based on an output of the application of the neural networkstructure to the captured sensor data at the block 2414.

If the determination at block 2416 results in a determination that anabandoned item is present in the automobile, then the process continuesat block 2420. At block 2420, an indicator is activated (e.g., audiblewarning tone emitted).

If the determination at block 2416 results in a determination that anabandoned item is not present in the automobile, then the processcontinues at block 2422. At block 2422, an indicator is not activated(e.g., audible warning tone not emitted).

FIG. 25 is a diagram of an exemplary neural network structure 2500according to some embodiments of the present disclosure. In someembodiments of the present disclosure, a neural network structure 2500may be used in a resource-constrained environment to categorize audiofiles or audio playback. For example, an audio file may be categorizedas “intense” or “not intense.”

The neural network structure 2500 includes layers 2552, 2554, 2556, and2558, which may be provided as described with respect to layers 652,654, 656, and 658, respectively. Neural network structure 2500 includesconnections 2562, 2564, and 2566, which may be provided as describedwith respect to the connections 662, 664, and 666, respectively.

In such embodiments, an audio file may be observed as a series of audioamplitudes, which may be represented by numeric values. As opposed tothe neural network structure 650 in which a series of pixel values wereprovided as input to input layer 652, the neural network structure 2500may receive the series of audio amplitudes as the input to the layer2552 (including nodes 2571, 2572, and 2573). Also, the output nodes mayinclude an “intense” category output node 2575 and a “not intense”category output node 2576. In this way, the neural network structure2500 may be trained to accurately categorize an audio file or audioplayback as “intense” or “not intense,” using the neural networkstructure training techniques described with respect to the neuralnetwork structure 650.

In some embodiments of the present disclosure, different categories thanthose just described for the neural network structure 2500 may be usedto categorize an audio file. For example, the neural network structure2500 may be trained to determine a an audio file category as one of thefollowing: “relaxed” and “aroused.”

FIGS. 26A and 26B are exemplary audio file formatting for input to aneural network structure according to some embodiments of the presentdisclosure. In some embodiments of the present disclosure, a neuralnetwork structure (e.g., neural network structure 2500) may be used tocategorize an audio file or audio playback. In such embodiments, it maybe advantageous to format the audio file as an image file for input to aneural network structure configured to receive image files as input.

In such embodiments, the audio file may be formatted as an image usingthe following technique. The audio amplitude of each sample of the audiofile may be converted to a pixel value (e.g., a grayscale value between0 and 255). This may be performed by determining the range of the pixelvalues (e.g., 0 to 255 for grayscale values) and the range of audioamplitude values (e.g., 0 to 100 for audio amplitude values) and mappingthe latter to the former in linear fashion (e.g., 0 audio amplitudevalue maps to 0 pixel value; 100 audio amplitude value maps to 255 pixelvalue; 50 audio amplitude maps to approximately 128 pixel value). Eachpixel value may then be stored serially as pixels in an image, startingwith the top-most and left-most pixel (traditionally the (0,0) pixel),moving to the right across the first row, then continue with theleft-most pixel on the second row, and so forth. FIG. 26A shows anexample image 2602 for an intense audio file, while FIG. 26B shows anexample image 2604 for a not intense audio file.

FIG. 27 is a flowchart of a process 2700 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure. For example, the resource-constrainedenvironment may be an automobile as described elsewhere in the presentdisclosure (e.g., automobiles 110, 300, 400, 710, 1510, and/or 1610).

At block 2702, sensor data is captured in a resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). For example, the sensor data may be captured by animage sensor located on the interior roof of the automobile. In suchembodiments, the image sensor may capture an overhead view image of theinterior of the automobile.

At block 2704, a configuration of the resource-constrained environmentis adjusted. In some embodiments, block 2704 may include changing aconfiguration of a physical object in the resource-constrainedenvironment. For example, block 2704 may include changing theforward/rearward position of the driver's seat, changing the level ofrecline of the driver's seat, changing forward/rearward position of thesteering wheel column, changing the forward/rearward position of apassenger's seat, changing the position of a sun visor, changing theopen/close position of air vents, etc.

At block 2706, a determination is made as to whether more configurationsare available for capturing sensor data. In some embodiments, block 2706may include determining whether there are more standard seat, steeringwheel, or other configuration for which sensor data has not beencaptured at a previous iteration of block 2704.

If the determination at block 2706 results in a determination that moreconfigurations are available for capturing sensor data, then the process2700 continues at block 2702.

If the determination at block 2706 results in a determination that moreconfigurations are not available for capturing sensor data, then theprocess 2700 continues at block 2708.

At block 2708, the sensor data is transferred from theresource-constrained environment to a non-resource-constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc.

At block 2710, a neural network structure is trained in thenon-resource-constrained environment using the transferred sensor data.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data transferredfrom the resource-constrained environment may be labelled prior to or aspart of block 2710.

At block 2712, the neural network structure is optimized. In someembodiments, block 2712 may include reducing a number of nodes in theneural network structure. In some embodiments, block 2712 may includereducing a number of connections in the neural network structure.

At block 2714, the neural network structure is transferred from thenon-resource-constrained environment to a resource constrainedenvironment. This transfer may be performed using any of a variety ofcommunications techniques, such as by a wired communication link, ashort-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2714 is a transfer to the same resource-constrained environmentfrom which the sensor data was transferred at block 2708 (e.g., to asame automobile, to a same mobile device, to a same embedded device). Insome embodiments, the transfer to the resource-constrained environmentat block 2714 is a transfer to a different resource-constrainedenvironment from which the sensor data was transferred at block 2708(e.g., to a different automobile, to a different mobile device, to adifferent embedded device).

At block 2716, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2716 is captured by the same one or more sensors used to capturesensor data at block 2702 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2716 iscaptured by a different one or more sensors used to capture sensor dataat block 2702 (e.g., a different video camera, a different infraredcamera). In some embodiments, the block 2716 is performed based on atrigger (e.g., a periodic schedule, a location of an automobile, turningoff an automobile).

At block 2718, the sensor data captured at block 2716 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2718 may include providing the sensor datacaptured at block 2716 as input to the neural network structure.

At block 2720, a determination is made as to whether the state of theresource-constrained environment represented in the sensor data capturedat block 2716 is a deviation from the state of the resource-constrainedenvironment reflected in the sensor data captured at block 2702.

If the determination at block 2720 results in a determination that adeviation is present in the sensor data captured at block 2716, then theprocess continues at block 2722. At block 2722, an action is performedbased on the determination that a deviation is present in theresource-constrained environment. For example, block 2722 may includeactivating an indicator, controlling the operation of an automobile,controlling the operation of an external computer system, and/or someother action.

If the determination at block 2720 results in a determination that nodeviation is present in the sensor data captured at block 2716, then theprocess continues at block 2724. At block 2724, no action is taken. Insome embodiments, no action is taken at block 2724 because no deviationis present in the resource-constrained environment. In some embodiments,some action may be taken at block 2724. In some embodiments, the actiontaken at block 2724 may include taking some action that is differentfrom the action performed at block 2722.

In some embodiments, the process 2700 may be used to takesafety-precaution actions in an automobile when a child or dog ispresent in an automobile. In such embodiments, block 2702 may includecapturing images of the interior of the automobile that reflect thestate of the automobile when no child or dog is present in theautomobile. For example, each iteration of the block 2702 may includecapturing an image of the interior of the automobile with the seat in adifferent position, the steering wheel in a different position, etc. Insuch instances, it may be appropriate to refer to the sensor datacaptured at block 2702 as a “baseline” for the state of the interior ofthe automobile. In such instances, it may also be appropriate to referto the sensor data captured at block 2702 as reflecting a “normal” or“empty” or “unoccupied” state of the interior of the automobile.

In such embodiments, the neural network structure trained at block 2710may reflect all “baseline” or “normal” configurations of the interior ofthe automobile. In such embodiments, the block 2720 may includedetermining that the sensor data captured at block 2716 represents adeviation from the “normal” or “baseline” state of the interior of theautomobile. In such instances, the determination at block 2720 may beperformed by determining whether the classification of the sensor dataapplied to the neural network structure at block 2718 results in ahighest classification confidence value below a predefined threshold(e.g., a confidence value below 25%).

In such embodiments, the block 2722 may include automatically causingthe automobile to open the windows (i.e., “roll them down”) to an openposition or the most-open position. In such embodiments, the block 2722may include automatically starting the automobile's engine and turningthe air conditioning to a “cool” setting. In such embodiments, the block2722 may include automatically activating the car alarm of theautomobile. In such embodiments, the block 2722 may include transmittinga message (e.g., sending an alert SMS message, sending an alert email,sending an alert notification through a software application) to thedriver of the automobile or another person. The message may includesensor data (e.g. an image of the interior). In some embodiments, theprocess 2700 may include only performing the action at block 2722 ifother conditions are met. For example, the action at block 2722 may beperformed only if the air temperature in the interior of the automobileis greater than or equal to 90 degrees Fahrenheit (e.g., as detected bya temperature sensor in the interior of the automobile).

In some embodiments, the process 2700 may be used to alert a driver whenan object is left in an automobile. In such embodiments, block 2702 mayinclude capturing images of the interior of the automobile that reflectthe state of the automobile when no object is left in the automobile. Insuch embodiments, an object may be considered “left” in the interior ofthe automobile if an object that is not typically present in theautomobile absent a driver (e.g., a wallet, a phone, a purse) is presentin the automobile when the driver is absent. In such embodiments, theiterations of the block 2702 may include capturing images of a“baseline” state of the interior of the automobile, as describedpreviously herein.

In such embodiments, the neural network structure trained at block 2710may reflect all “baseline” configurations of the interior of theautomobile. In such embodiments, the block 2720 may include determiningthat the sensor data captured at block 2716 represents a deviation fromthe “baseline” state of the interior of the automobile. In suchinstances, the determination at block 2720 may be performed bydetermining whether the classification of the sensor data applied to theneural network structure at block 2718 results in a highestclassification confidence value below a predefined threshold (e.g., aconfidence value below 25%).

In such embodiments, the block 2722 may include automatically alerting adriver of the automobile of the presence of the item left in theautomobile. In such embodiments, the block 2722 may includeautomatically activating an indicator in the automobile (e.g., a chimeor light). In such embodiments, the block 2722 may include automaticallyactivating the car alarm of the automobile. In such embodiments, theblock 2722 may include transmitting a message (e.g., sending an alertSMS message, sending an alert email, sending an alert notificationthrough a software application) to the driver of the automobile oranother person. The message may include sensor data (e.g. an image ofthe interior).

In such embodiments, the process 2700 may include only performing theaction at block 2722 when certain conditions are met. For instance, theaction performed at block 2722 may only be performed after the driver ofthe automobile locks the automobile. As another instance, the actionperformed at block 2722 may only be performed after a predefined time ofday (e.g., 20:00). As another instance, the action performed at block2722 may only be performed when the automobile is present in apredefined location (e.g., located in the garage at the house of thedriver of the automobile). As another instance, the action performed atblock 2722 may only be performed after a predefined action by the driverof the automobile (e.g., the driver returning the automobile to a rentalcompany). As another instance, the action performed at block 2722 mayonly be performed after being triggered by a user (e.g., triggered by aninput to a computer system by an employee of a rental car company). Insome embodiments, the process 2700 may include only performing block2716, block 2718, block 2720, and block 2722 when one or more of theaforementioned conditions are met.

In some embodiments, the process 2700 may be used to prevent a rentalcar driver from returning an automobile. In such embodiments, block 2702may include capturing images of the interior of the automobile thatreflect the state of the automobile when no object is left in theautomobile and no damage has been done to the interior of the automobile(e.g., no stains on the seats). In such embodiments, the iterations ofthe block 2702 may include capturing images of a “baseline” state of theinterior of the automobile, as described previously herein. In someembodiments, the images of the “baseline” state of the interior of theautomobile may reflect some changes from the normal configuration of theinterior of the automobile. For example, if a previous driver of theautomobile left a stain on the driver's seat of the automobile, then theimages of the “baseline” state of the interior of the automobile mayreflect this damage to the driver's seat. As another example, if thepresent driver of the automobile is using a child safety seat as part ofoperation of the automobile (e.g., having rented the child safety seatwith rental of the automobile from a rental car company), then theimages of the “baseline” state of the interior of the automobile mayreflect this presence of the child safety seat.

In such embodiments, the neural network structure trained at block 2710may reflect all “baseline” configurations of the interior of theautomobile. In such embodiments, the block 2720 may include determiningthat the sensor data captured at block 2716 represents a deviation fromthe “baseline” state of the interior of the automobile. In suchinstances, the determination at block 2720 may be performed bydetermining whether the classification of the sensor data applied to theneural network structure at block 2718 results in a highestclassification confidence value below a predefined threshold (e.g., aconfidence value below 25%).

In such embodiments, the block 2722 may include preventing a driver ofthe automobile from returning the automobile to a car rental company orservice. For example, if the determination at block 2720 determines thata deviation is present in the interior of automobile, then the actionperformed at block 2722 may include controlling an external softwareapplication so that the driver of the automobile cannot set the state ofthe automobile to “returned” or “checked-in” in a software application(e.g., preventing a car sharing user from setting a shared car status to“available”). As another example, if the determination at block 2720determines that a deviation is present in the interior of automobile,then the action performed at block 2722 may include preventing thedriver of the automobile from entering a car return parking structure.For instance, a rental car company may maintain a parking lot, parkinggarage, or other structure for returning rental cars, where the parkingstructure is not supervised by a human employee. But entry to theparking structure may be controlled by a parking gate, retractablepylons, or some other automatically controlled structures. In such anexample, the action performed at block 2722 may include preventing theparking gate form opening, the pylons from retracting, or the like.

In such embodiments, the process 2700 may include only performing theaction at block 2722 when certain conditions are met. For instance, theaction performed at block 2722 may only be performed after the driverparks a car share automobile in a designated car share parking spot. Asanother instance, the action performed at block 2722 may only beperformed after the driver parks a car share automobile in a designatedcar share parking spot, exits the automobile, and locks the automobile.As another instance, the action performed at block 2722 may only beperformed after the driver of a car share automobile touches a “checkin” button (or something similar) in a car share software application ona mobile device. As another instance, the action performed at block 2722may only be performed after the driver of a rental car automobile parksthe automobile outside the automobile return parking structure. Asanother instance, the action performed at block 2722 may only beperformed after the driver of a rental car automobile touches a “return”button (or something similar) in a rental car software application on amobile device. In some embodiments, the process 2700 may include onlyperforming block 2716, block 2718, block 2720, and block 2722 when oneor more of the aforementioned conditions are met.

In some embodiments, the process 2700 may be used to process a rentalcar after being returned by a renter. In such embodiments, block 2702may include capturing images of the interior of the automobile thatreflect the state of the automobile when no object is left in theautomobile and no damage has been done to the interior of the automobile(e.g., no stains on the seats). In such embodiments, the iterations ofthe block 2702 may include capturing images of a “baseline” state of theinterior of the automobile, as described previously herein. In someembodiments, the images of the “baseline” state of the interior of theautomobile may reflect some changes from the normal configuration of theinterior of the automobile. For example, if a previous driver of theautomobile left a stain on the driver's seat of the automobile, then theimages of the “baseline” state of the interior of the automobile mayreflect this damage to the driver's seat. As another example, if thepresent driver of the automobile is using a child safety seat as part ofoperation of the automobile (e.g., having rented the child safety seatwith rental of the automobile from a rental car company), then theimages of the “baseline” state of the interior of the automobile mayreflect this presence of the child safety seat.

In such embodiments, the neural network structure trained at block 2710may reflect all “baseline” configurations of the interior of theautomobile. In such embodiments, the block 2720 may include determiningthat the sensor data captured at block 2716 represents a deviation fromthe “baseline” state of the interior of the automobile. In suchinstances, the determination at block 2720 may be performed bydetermining whether the classification of the sensor data applied to theneural network structure at block 2718 results in a highestclassification confidence value below a predefined threshold (e.g., aconfidence value below 25%).

In such embodiments, the block 2722 may include performing actionsassociated with processing a rental car automobile after being returnedfrom a renter. For example, the action performed at block 2722 mayinclude automatically generating an action item list for processing theautomobile in order to return it to a rental-ready state. In such anexample, block 2722 may include adding an item “vacuum center console”,“return driver seat back to upright position”, and/or “remove childsafety seat” if any or all of these actions are necessary to return theautomobile to a standard state for renting to another customer of therental car company. In such examples, block 2722 may include generatingan action item list and presenting the action item list to an employeeof the rental car company (e.g., printing the list on a printer in a carservice bay, transmitting the list for presentation on a mobile deviceof a car service technician). As another example, block 2722 may includeautomatically generating a financial charge for the renter of theautomobile (e.g., based on damage to the interior of the automobile).

In such embodiments, the process 2700 may include only performing theaction at block 2722 when certain conditions are met. For instance, theaction performed at block 2722 may only be performed after a driver of arental automobile parks the automobile in an automobile return parkingstructure. In some embodiments, the process 2700 may include onlyperforming block 2716, block 2718, block 2720, and block 2722 when oneor more of the aforementioned conditions are met.

In some embodiments, the process 2700 may include more than one neuralnetwork structure. For example, block 2702 and block 2704 may beiterated in order to capture multiple sensor data for a first baseline,which then may be used to train a first neural network structure atblock 2710. Continuing the example, block 2702 and block 2704 may befurther iterated in order to capture multiple sensor data for a secondbaseline, which then may be used to train a second neural networkstructure at block 2710. In such an example, the process 2700 mayinclude selecting one of the more than one neural network structureprior to block 2718, and then applying the sensor data captured at block2716 to the selected neural network structure at block 2718.

For instance, a first neural network structure may be trained for a “nochild safety seat” baseline, and a second neural network structure maybe trained for a “rear-facing child safety seat in left-rear passengerseat” baseline. The first neural network structure may be trained atblock 2710 by using sensor data captured at block 2702 over multipleconfiguration of an automobile interior with no child safety seatpresent. For instance, the first configuration could include no childsafety seat present and the driver's seat at the rear-most position. Asecond configuration could include no child safety seat present and thedriver's seat at position slightly forward from the rear-most position.And so on. The second neural network structure may be trained at block2710 by using sensor data captured at block 2702 over multipleconfiguration of an automobile interior with a rear-facing child safetyseat in the left-rear passenger seat. For instance, the firstconfiguration could include a rear-facing child safety seat in theleft-rear passenger seat and the driver's seat at the rear-mostposition. A second configuration could include a rear-facing childsafety seat in the left-rear passenger seat and the driver's seat atposition slightly forward from the rear-most position.

In such an instance, process 2700 may include a determination prior toblock 2718 of whether the first neural network structure or the secondneural network structure should be used at block 2718. For instance, ifthe automobile is a rental car and the current renter did not rent achild safety seat, then the first neural network structure (i.e., “nochild safety seat”) may be used at block 2718. If the automobile is arental car and the current renter rented a rear-facing child safetyseat, then the second neural network structure (i.e., “rear-facing childsafety seat in left-rear passenger seat”) may be used at block 2718.

In such embodiments, additional information may be used to determinewhich neural network to use at block 2718. For instance, an externalcomputing system may be queried to determine whether a renter of theautomobile has rented a child safety seat, what kind of safety seat,and/or where the child safety seat was installed by the servicetechnician.

In some embodiments, generic and customized neural network structuresmay be used. For example, a generic “2016 Chevrolet Sonic hatchback”neural network structure may be trained at block 2710 using sensor datafrom one or more 2016 Chevrolet Sonic hatchbacks. The generic “2016Chevrolet Sonic hatchback” neural network may then be used at block 2718in multiple 2016 Chevrolet Sonic hatchbacks. Continuing the example, anautomobile that has a unique deviation from a generic configuration mayuse a customized neural network structure. For example, if a 2016Chevrolet Sonic hatchback with VIN=12341234123412345 has a large stainon the driver's seat, then a customized “VIN 12341234123412345” neuralnetwork structure may be trained at block 2710 using sensor data fromthat specific vehicle. The customized “VIN 12341234123412345” neuralnetwork structure may then be used at block 2718 in that specificvehicle. In such embodiments, the block 2702, block 2704, block 2706,block 2708, block 2710, and block 2712 may be omitted for a new vehiclefor which a generic neural network structure already exists. In suchembodiments, the process 2700 may include transferring a generic neuralnetwork structure to the automobile at block 2714 unless a customizedneural network structure exists for the automobile, in latter case whichthe customized neural network structure may be transferred to theautomobile at block 2714.

In various embodiments, process 2700 may be performed using structuresdescribed elsewhere herein. For example, blocks 2702 and 2716 may beperformed using sensors described elsewhere herein (e.g., sensors 460,472, 716, 765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616,1618, 1716). As a further example, blocks 2708, 2718, and 2720 may beperformed using computing devices described elsewhere herein (e.g.,computing devices 726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As afurther example, blocks 2710, 2712, and 2714 may be performed usingcomputing devices described elsewhere herein (e.g., computing devices720, 770, 1000, 1320, 1420, 1520, 1620, 1720). As a further example,blocks 2722 and 2724 may be performed using described elsewhere herein(e.g., controllers 728, 778, 1200, 1328, 1428, 1528, 1628, 1728).

FIG. 28 is a flowchart of a process 2800 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure. For example, the resource-constrainedenvironment may be an automobile as described elsewhere in the presentdisclosure (e.g., automobiles 110, 300, 400, 710, 1510, and/or 1610).

At block 2802, sensor data is captured of baseline states in aresource-constrained environment. The sensor data may be captured by anyof a variety of sensors, such as described with respect to sensorselsewhere in the present disclosure (e.g., sensors 232, 234, 460, 472,716, 800, 1516, 1518, 1616, 1716). For example, the sensor data may becaptured by an image sensor located on the interior roof of theautomobile. In such embodiments, the image sensor may capture anoverhead view image of the interior of the automobile. The sensor datacaptured at block 2802 may reflect one or more baseline states of theautomobile as described elsewhere herein. For example, the sensor datacaptured at block 2802 may include images of an interior of anautomobile when no people or abandoned objects are present. In someembodiments, block 2802 may be performed iteratively across multiplebaseline state configuration of the resource-constrained environment,such as described for block 2702, block 2704, and block 2706 of process2700 in FIG. 27.

At block 2814, sensor data is captured of deviation states in aresource-constrained environment. The sensor data may be captured by anyof a variety of sensors, such as described with respect to sensorselsewhere in the present disclosure (e.g., sensors 232, 234, 460, 472,716, 800, 1516, 1518, 1616, 1716). For example, the sensor data may becaptured by an image sensor located on the interior roof of theautomobile. In such embodiments, the image sensor may capture anoverhead view image of the interior of the automobile.

The sensor data captured at block 2814 may reflect one or more deviationstates of the automobile. Deviation states may include situations wherethe state of the interior of the automobile deviates from the one ormore baseline states described elsewhere herein. For example, block 2814may include capturing images of the interior of the automobile when asmartphone (i.e., a “left” or “abandoned” item) is present on thedriver's seat. As another example, block 2814 may include capturingimages of the interior of the automobile when a cup (i.e., a “left” or“abandoned” item) is present in a cup holder of the center console ofthe automobile. As another example, block 2814 may include capturingimages of the interior of the automobile when an umbrella (i.e., a“left” or “abandoned” item) is present in a passenger footwell of theautomobile. As another example, block 2814 may include capturing imagesof the interior of the automobile when a food wrapper (i.e., trash orother refuse) is present in a passenger footwell of the automobile. Asanother example, block 2814 may include capturing images of the interiorof the automobile when a stain (i.e., damage) is present on a driverseat of the automobile. As another example, block 2814 may includecapturing images of the interior of the automobile when a cigarette burn(i.e., damage) is present on a passenger seat of the automobile. In someembodiments, block 2814 may be performed iteratively across multipledeviation state configuration of the resource-constrained environment.

At block 2804, a neural network structure is trained in thenon-resource-constrained environment using the sensor data captured atblock 2802 and the sensor data captured at block 2814. The training ofthe neural network structure may be performed as described elsewhereherein (e.g., as described for neural network structures 600, 650). Insome embodiments, the sensor data captured from the resource-constrainedenvironment may be labelled prior to or as part of block 2804. Forexample, the sensor data captured at block 2802 may be labelled“baseline,” “normal,” or the like. The sensor data captured at block2814 may be labelled “deviation,” “abnormal,” or the like. In someembodiments, block 2804 may be performed after first transferring thesensor data captured at block 2802 and/or block 2814 to a non-resourceconstrained environment, such as described for block 2708 of process2700 in FIG. 27.

At block 2806, the neural network structure trained at block 2804 istransferred from the non-resource-constrained environment to a resourceconstrained environment. This transfer may be performed using any of avariety of communications techniques, such as by a wired communicationlink, a short-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2806 is a transfer to the same resource-constrained environmentfrom which the sensor data was captured at block 2802 and/or block 2814(e.g., to a same automobile, to a same mobile device, to a same embeddeddevice). In some embodiments, the transfer to the resource-constrainedenvironment at block 2806 is a transfer to a differentresource-constrained environment from which the sensor data was capturedat block 2802 and/or block 2814 (e.g., to a different automobile, to adifferent mobile device, to a different embedded device). In someembodiments, block 2812 may be performed after first optimizing theneural network structure trained at block 2804, such as described forblock 2712 of process 2700 in FIG. 27.

At block 2808, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2808 is captured by the same one or more sensors used to capturesensor data at block 2802 and/or block 2814 (e.g., a same video camera,a same infrared camera). In some embodiments, the sensor data capturedat block 2808 is captured by a different one or more sensors used tocapture sensor data at block 2802 and/or block 2814 (e.g., a differentvideo camera, a different infrared camera). In some embodiments, theblock 2808 is performed based on a trigger (e.g., a periodic schedule, alocation of an automobile, turning off an automobile).

At block 2810, the sensor data captured at block 2808 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2810 may include providing the sensor datacaptured at block 2808 as input to the neural network structure.

At block 2812, a determination is made as to whether the state of theresource-constrained environment represented in the sensor data capturedat block 2808 is a deviation from the baseline state of theresource-constrained environment reflected in the sensor data capturedat block 2802. In some embodiments, the determination at block 2812 maybe performed by determining whether the output of block 2810 produces alabel with a highest confidence value of “baseline” or a label with ahighest confidence value of “deviation.”

If the determination at block 2812 results in a determination that adeviation is present in the sensor data captured at block 2808, then theprocess continues at block 2820.

At block 2816, a neural network structure is trained in thenon-resource-constrained environment using the sensor data captured atblock 2814. The training of the neural network structure may beperformed as described elsewhere herein (e.g., as described for neuralnetwork structures 600, 650). In some embodiments, the sensor datacaptured from the resource-constrained environment may be labelled priorto or as part of block 2816. For example, the sensor data captured atblock 2814 may be labelled “left phone”, “left phone on driver's seat”,“left phone in center console”, “left cup”, “left cup in centerconsole”, “left item in footwell”, “left item in passenger footwell”,“left umbrella in front passenger footwell”, “refuse”, “left refuse inpassenger footwell”, “left food wrapper in passenger footwell”, “foodcrumbs in passenger footwell”, “stain on driver's seat”, “large stain ondriver's seat”, “small stain on driver's seat”, “fabric damage”, “burnmark on passenger seat”, or the like. In some embodiments, block 2816may be performed after first transferring the sensor data captured atblock 2814 to a non-resource constrained environment, such as describedfor block 2708 of process 2700 in FIG. 27.

At block 2818, the neural network structure trained at block 2816 istransferred from the non-resource-constrained environment to a resourceconstrained environment. This transfer may be performed using any of avariety of communications techniques, such as by a wired communicationlink, a short-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2818 is a transfer to the same resource-constrained environmentfrom which the sensor data was captured at block 2814 (e.g., to a sameautomobile, to a same mobile device, to a same embedded device). In someembodiments, the transfer to the resource-constrained environment atblock 2818 is a transfer to a different resource-constrained environmentfrom which the sensor data was captured at block 2814 (e.g., to adifferent automobile, to a different mobile device, to a differentembedded device). In some embodiments, block 2818 may be performed afterfirst optimizing the neural network structure trained at block 2816,such as described for block 2712 of process 2700 in FIG. 27.

At block 2820, the sensor data captured at block 2808 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 2820 may include providing the sensor datacaptured at block 2808 as input to the neural network structure. In someembodiments, block 2808 may include determining a type of deviation inthe resource-constrained environment based on the output of applying thesensor data captured at block 2808 to the neural network structuretrained at block 2816. For example, the type of deviation present may bedetermined as the output label for the neural network with the highestconfidence value for the input sensor data. In such embodiments, adefault deviation type (e.g., “unidentified deviation”) may bepredefined for cases where no output label has a confidence value abovea predefined threshold (e.g., no confidence value above 25%).

At block 2822, an action is performed based on the type of deviationthat is determined to be present in the resource-constrained environmentat block 2820. For example, if the deviation that is determined to bepresent is the presence of refuse in the automobile, then the actionperformed at block 2822 may include sending an alert (e.g., message in asoftware application on a mobile device) to a car service technicianthat is an employee of a rental car company that provides the automobileas a rental vehicle. As another example, if the deviation that isdetermined to be present is the presence of a wallet in the automobile,then the action performed at block 2822 may include sending an alert(e.g., a text message) to an owner or most recent renter of theautomobile. As another example, if the deviation that is determined tobe present is the presence of a stain in the automobile, then the actionperformed at block 2822 may include generating a charge to a credit cardassociated with a most recent renter of the automobile. In someembodiments, block 2822 may include performing any of the actionsdescribed for block 2722 of process 2700 in FIG. 27. Other deviationtype-specific actions may be defined and performed at block 2822 inaccordance with embodiments of the present disclosure.

If the determination at block 2812 results in a determination that nodeviation is present in the sensor data captured at block 2808, then theprocess continues at block 2824. At block 2824, no action is taken. Insome embodiments, no action is taken at block 2824 because no deviationis present in the resource-constrained environment. In some embodiments,some action may be taken at block 2824. In some embodiments, the actiontaken at block 2824 may include taking some action that is differentfrom the action performed at block 2822.

In some embodiments, process 2800 may be modified to not use twoseparate neural network structures. For example, for the sensor datacaptured at block 2814, the labels associated therewith for the trainingat block 2816 may be provided as labels for the training at block 2804.In such an example, block 2816, block 2818, and/or block 2820 may beomitted entirely. In such an example, the determination as to the typeof deviation that is present may be made as part of block 2810. And insuch an example, the type of deviation determined at block 2810 may beused at block 2822 to determine the action to perform.

In various embodiments, process 2800 may be performed using structuresdescribed elsewhere herein. For example, blocks 2802, 2808, and 2814 maybe performed using sensors described elsewhere herein (e.g., sensors460, 472, 716, 765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518,1616, 1618, 1716). As a further example, blocks 2810, 2812, 2820, and2822 may be performed using computing devices described elsewhere herein(e.g., computing devices 726, 776, 1100, 1316, 1416, 1526, 1626, 1726).As a further example, blocks 2804, 2806, 2816, and 2818 may be performedusing computing devices described elsewhere herein (e.g., computingdevices 720, 770, 1000, 1320, 1420, 1520, 1620, 1720). As a furtherexample, blocks 2822 and 2824 may be performed using described elsewhereherein (e.g., controllers 728, 778, 1200, 1328, 1428, 1528, 1628, 1728).

FIG. 29 is a flowchart of a process 2900 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure. For example, the resource-constrainedenvironment may be an automobile as described elsewhere in the presentdisclosure (e.g., automobiles 110, 300, 400, 710, 1510, and/or 1610).

At block 2902, sensor data is captured of baseline states in aresource-constrained environment. The sensor data may be captured by anyof a variety of sensors, such as described with respect to sensorselsewhere in the present disclosure (e.g., sensors 232, 234, 460, 472,716, 800, 1516, 1518, 1616, 1716). For example, the sensor data may becaptured by an image sensor located on the interior roof of theautomobile. In such embodiments, the image sensor may capture anoverhead view image of the interior of the automobile. The sensor datacaptured at block 2902 may reflect one or more baseline states of theautomobile as described elsewhere herein. For example, the sensor datacaptured at block 2902 may include images of an interior of anautomobile when no people or abandoned objects are present. In someembodiments, block 2902 may be performed iteratively across multiplebaseline state configuration of the resource-constrained environment,such as described for block 2702, block 2704, and block 2706 of process2700. For example, block 2902 may be performed iteratively usingbaseline configuration with varying seat positions, lighting modes, andother physical configurations that represent various normal, cleanautomobile interior configurations.

At block 2904, a neural network structure is trained in thenon-resource-constrained environment using the sensor data captured atblock 2904. The training of the neural network structure may beperformed as described elsewhere herein (e.g., as described for neuralnetwork structures 600, 650). In some embodiments, the sensor datacaptured from the resource-constrained environment may be labelled priorto or as part of block 2904. For example, the sensor data captured atblock 2802 may be labelled “baseline,” “normal,” or the like. In someembodiments, block 2904 may be performed after first transferring thesensor data captured at block 2902 to a non-resource constrainedenvironment, such as described for block 2708 of process 2700 in FIG.27.

At block 2906, the neural network structure trained at block 2904 istransferred from the non-resource-constrained environment to a resourceconstrained environment. This transfer may be performed using any of avariety of communications techniques, such as by a wired communicationlink, a short-range wireless communication link, a long-range wirelesscommunication link, a transfer of physical storage media, etc. In someembodiments, the transfer to the resource-constrained environment atblock 2906 is a transfer to the same resource-constrained environmentfrom which the sensor data was captured at block 2902 (e.g., to a sameautomobile, to a same mobile device, to a same embedded device). In someembodiments, the transfer to the resource-constrained environment atblock 2906 is a transfer to a different resource-constrained environmentfrom which the sensor data was captured at block 2902 (e.g., to adifferent automobile, to a different mobile device, to a differentembedded device). In some embodiments, block 2912 may be performed afterfirst optimizing the neural network structure trained at block 2904,such as described for block 2712 of process 2700 in FIG. 27.

At block 2908, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments, the sensor data captured atblock 2908 is captured by the same one or more sensors used to capturesensor data at block 2902 (e.g., a same video camera, a same infraredcamera). In some embodiments, the sensor data captured at block 2908 iscaptured by a different one or more sensors used to capture sensor dataat block 2902 (e.g., a different video camera, a different infraredcamera). In some embodiments, the block 2908 is performed based on atrigger (e.g., a periodic schedule, a location of an automobile, turningoff an automobile).

At block 2910, a model is generated based on the neural networkstructure trained at block 2906 and based on the sensor data captured atblock 2908. In some embodiments, the model generated at block 2910 maybe an image generated based on the neural network structure trained atblock 2906 and based on the sensor data captured at block 2908. In suchembodiments, the generated image may be an image generated that issimilar to the sensor data captured at block 2908 but that is generatedusing the neural network structure trained at block 2904 (and thus thesensor data captured at block 2902). The model may be generated using agenerative adversarial network and/or other neural network technique.

For example, block 2910 may include providing the sensor data capturedat block 2908 as input to the neural network structure trained at block2904 and prompting the neural network structure 2904 to generate sensordata that is the most similar to the sensor data captured at block 2908as possible. For instance, in the case of images of the interior of anautomobile, the block 2910 may include prompting the neural networkstructure trained at block 2904 to generate an image of the interior ofthe automobile that is as similar as possible to the image of theinterior of the automobile captured at block 2908. In cases where theneural network structure is trained at block 2904 using only images ofthe interior of the automobile captured at block 2902 that representnormal, clean configurations, and in cases where the image captured atblock 2908 includes a portion that deviates from those normal, cleanconfigurations (e.g., stain on driver's seat, missing floor mat, cup incupholder, trash on floor), the neural network structure may be able togenerate an image at block 2910 that approximates the image captured atblock 2908 but that cannot generate the deviant aspect (e.g., the stain,the lack of floor mat, the cup, the trash) because the images on whichthe neural network structure was trained at block 2904 did not containany such aspects.

At block 2912, a difference is calculated between the sensor datacaptured at block 2908 and the model generated at block 2910. Forexample, block 2912 may include generating an array or matrixrepresentation of the sensor data and an array or matrix representationof the model. In such an example, each element of the array or matrixmay be a pixel value for the sensor data and model, respectively. Thedifference may be calculated by performing an elementwise subtraction ofthe pixel values. The resulting difference may be an array or matrixrepresentation of an image, wherein each element is a pixel value.

At block 2914, a determination is made as to whether a deviation ispresent in sensor data. Block 2914 may include determining whether adeviation exists in the sensor data captured at block 2908 with respectto the baseline state of the resource-constrained environment reflectedin the sensor data captured at block 2902. For example, block 2914 mayinclude determining whether there is at least one pixel value in thedifference calculated at block 2912 that is greater than a predefinedthreshold (e.g., greater than 64 on a 0 to 256 pixel value scale). Asanother example, block 2914 may include determining whether there is acluster of pixels (e.g., 16 adjacent pixels) with each having a pixelvalue in the difference calculated at block 2912 that is greater than apredefined threshold (e.g., each pixel value greater than 64 on a 0 to256 pixel value scale).

If the determination at block 2914 results in a determination that nodeviation is present in the sensor data captured at block 2908, then theprocess continues at block 2918. At block 2918, no action is taken. Insome embodiments, no action is taken at block 2918 because no deviationis present in the resource-constrained environment. In some embodiments,some action may be taken at block 2918. In some embodiments, the actiontaken at block 2918 may include taking some action that is differentfrom the action performed at block 2916.

If the determination at block 2914 results in a determination that adeviation is present in the sensor data captured at block 2908, then theprocess continues at block 2916. At block 2916, an action is performedbased on the determination that a deviation is present in theresource-constrained environment. For example, block 2916 may includeactivating an indicator, controlling the operation of an automobile,controlling the operation of an external computer system, and/or someother action. In some embodiments, block 2914 may include performing anyof the actions described for block 2722 of process 2700 in FIG. 27and/or any of the actions described for block 2822 of process 2800 inFIG. 28.

In some embodiments, process 2900 may include determining a location ofa deviation in the resource-constrained environment. For example, ifblock 2914 results in a determination that deviation is present in theresource-constrained environment based on a cluster of high pixel valuesin a difference array (as calculated at block 2912), then process 2900may further include determining a location of the deviation. Continuingthe example, the process 2900 may include determining a center of thecluster of high pixel values (e.g., arithmetic mean of x and ycoordinates). Continuing the example, the process 2900 may includeidentifying a location in which the center of the cluster is located.For instance, different zones (e.g., rectangles each defined by four x,y coordinates) may be defined on image area (e.g., driver's seat, centerconsole, passenger seat, dashboard, etc.). In such an instance, thelocation of the deviation may be determined by determining in which zonethe center of the cluster is located. In some embodiments, the locationdetermined for the deviation may be used to determine what action totake at block 2916.

In some embodiments, the process 2900 may include determining the typeof deviation that is present in the resource constrained environment.For example, a second neural network may be trained using only images ofdeviations (e.g., image of a smartphone, image of a wallet, image of astained seat). When a deviation is determined to be present in theresource-constrained environment at block 2914, the process 2900 mayinclude cropping the sensor data captured at block 2908 to tightly boundthe detected deviation (e.g., tightest bounding rectangle possiblearound the cluster of high pixel values in the calculated differencewithout excluding any of the high pixel values). Continuing the example,the cropped sensor data may be provided as input to the second neuralnetwork, the output of which (e.g., a highest confidence deviationlabel) may be used to determine the deviation present in theresource-constrained environment). In such embodiments, when more thanone cluster of high pixel values exist in the calculated difference, thedeviation-identification approach just described may be repeated foreach cluster.

According to the foregoing description of process 2900, including theaforementioned modifications thereto, the process 2900 may includeoutputting a description of the detected deviation and its location atblock 2916. For example, the aforementioned techniques may be used todetermine that the location of the deviation is “food wrapper” and thelocation is “driver footwell.” In such an example, the process 2900 mayoutput “food wrapper in driver footwell” at block 2916 (with or withoutaddition actions). As another example, the aforementioned techniques maybe used to determine that the location of a first deviation is “cup”with a location of “center console,” and a second deviation of “stain”with a location of “front passenger seat.” In such an example, theprocess 2900 may output “(1) cup in center console; (2) stain on frontpassenger seat” at block 2916 (with or without addition actions).

In various embodiments, process 2900 may be performed using structuresdescribed elsewhere herein. For example, blocks 2902 and 2908 may beperformed using sensors described elsewhere herein (e.g., sensors 460,472, 716, 765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616,1618, 1716). As a further example, blocks 2910, 2912, 2914, 2916, and2918 may be performed using computing devices described elsewhere herein(e.g., computing devices 726, 776, 1100, 1316, 1416, 1526, 1626, 1726).As a further example, blocks 2904, 2906, 2910, 2912, 2914, 2916, and2918 may be performed using computing devices described elsewhere herein(e.g., computing devices 720, 770, 1000, 1320, 1420, 1520, 1620, 1720).As a further example, blocks 2916 and 2918 may be performed usingdescribed elsewhere herein (e.g., controllers 728, 778, 1200, 1328,1428, 1528, 1628, 1728).

FIGS. 30 and 31 are diagrams of sensor data and difference calculationsaccording to some embodiments of the present disclosure. Image 3002 isan example of sensor data captured of a resource-constrained environment(e.g., as captured at block 2716, block 2808, block 2908). Image 3004 isan example of baseline sensor data captured of a resource-constrainedenvironment (e.g., as captured at block 2702, block 2802, block 2902).Image 3002 shows a deviation from the baseline state of the automobileinterior. Specifically, image 3002 includes a smartphone present in thedriver's seat.

Image 3006 is a difference calculated between image 3002 and image 3004.In image 3006, the minimum difference (i.e., no difference) between apixel in image 3002 and image 3004 is shown in black. The maximumdifference between a pixel in image 3002 and image 3004 is shown inwhite. Intermediate differences in pixel values are shown in othercolors. Image 3006 contains a cluster of high pixel values at thelocation of the smartphone, which accurately identifies the presence ofthe deviation (i.e., the smartphone). However, the image 3006 alsocontains many non-zero pixel values elsewhere, which inaccuratelyidentify deviations where there are none. As such, image 3002, image3004, and image 3006 illustrate that a difference calculation based on asingle baseline image of the resource-constrained environment may tendto be error prone.

Image 3102 is an example of sensor data captured of aresource-constrained environment (e.g., as captured at block 2716, block2808, block 2908). Image 3104 is an example of a model generated of aresource-constrained environment (e.g., as generated at block 2910).Image 3102 shows a deviation from the baseline state of the automobileinterior. Specifically, image 3002 includes a smartphone present in thedriver's seat.

Image 3106 is a difference calculated between image 3102 and model 3104.In image 3106, the pixel difference values are shown as previouslydescribed for image 3006. Image 3106 contains a cluster of high pixelvalues at the location of the smartphone, which accurately identifiesthe presence of the deviation (i.e., the smartphone). In contrast toimage 3006, image 3106 does not contain any other high pixel values,which accurately identifies that there are no other deviations in image3102.

FIG. 32 is a flowchart of a process 3200 for employing a neural networkin a resource-constrained environment according to some embodiments ofthe present disclosure. For example, the resource-constrainedenvironment may be an automobile as described elsewhere in the presentdisclosure (e.g., automobiles 110, 300, 400, 710, 1510, and/or 1610). Insome embodiments, the process 3200 may be performed after havingcaptured sensor data in a resource constrained environment (e.g., block2702, 2802, 2902) and after having trained a neural network structurebased on that sensor data (e.g., block 2710, 2804, 2904).

At block 3202, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 3204, the sensor data captured at block 3202 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, block 3204 may include providing the sensor datacaptured at block 3202 as input to the neural network structure.

At block 3206, output data of the neural network structure is stored.Block 3206 may include storing a classification and confidence levelgenerated by applying the sensor data as input to the neural networkstructure at block 3204. For example, if the result of block 3204 is aprediction of “touching head unit” with a confidence level of 90%, then“touching head unit” and 90% are stored at block 3206. The neuralnetwork structure output data stored at block 3206 may be referred to as“original” neural network structure output data in this context, becausethe output data is the output of the neural network structure prior toany masking activity described hereinafter.

At block 3208, a mask is applied to the sensor data captured at block3202. A mask may include deleting a portion of the actual sensor dataand replacing it with default value sensor data. For example, if thesensor data is an image (e.g., with each pixel value on a 0 to 256grayscale), then a mask may be a rectangle (e.g., 4 pixels by 4 pixels)of all-black pixels (pixel value of 0). In this example, placing themask on the sensor data at block 3208 may include replacing thetop-rightmost 4×4 rectangle of pixels with the mask of all-black pixels.

At block 3210, the masked sensor data generated at block 3208 is appliedto the neural network structure. The neural network structure may be thesame neural network structure used at block 3204. Block 3210 may resultin generating new neural network structure output data reflecting thenew input (i.e., the masked sensor data). For example, block 3210 mayresult in generating a new prediction (e.g., “touching head unit”) witha new confidence level (e.g., 91%). The new class prediction may be thesame or different from the original neural network structure outputdata. The new confidence level may be the same or different from theoriginal neural network structure output data.

At block 3212, a determination is made as to whether block 3210 resultedin different output data than the original neural network structureoutput data. For example, block 3212 may include determining whether thenew class prediction is different from the class prediction in thestored original neural network structure output data. Continuing theexample, block 3212 may include determining whether the new confidencelevel is different from the class prediction in the stored originalneural network structure output data. In some embodiments, block 3212may include determining whether the new output data is different fromthe original output data by a sufficient minimum threshold. For example,the determination at block 3212 may result in a determination of “yes”if either the class prediction has changed or the confidence level haschanged by 10% (either up or down).

If the determination at block 3212 results in a determination that block3210 resulted in different neural network output data, then the processcontinues at block 2914. At block 3214, the mask location is stored.Block 3214 may include storing the location of all pixels covered by themask applied at the most recent iteration of block 3208. For example,block 3214 may include storing the four x, y coordinates of therectangular mask used at the most recent iteration of block 3208.

If the determination at block 3212 results in a determination that block3210 resulted in different neural network output data, then the processcontinues at block 3216.

At block 3216, a determination is made as to whether any more locationsare available for placement of the mask. For example, block 3216 mayinclude determining whether any pixels in the sensor data captured atblock 3202 have not yet been covered by the mask in an iteration ofblock 3208. As another example, block 3216 may include determiningwhether any pixel in the sensor data captured at block 3202 has not beencovered by the mask in at least four iterations of block 3208. Asanother example, block 3216 may include determining whether any pixel inthe sensor data captured at block 3202 has not yet been the centerposition of the mask in a previous iteration of block 3208.

If the determination at block 3216 results in a determination that thereare more mask locations available, then process 3200 continues at block3208.

If the determination at block 3216 results in a determination that thereare no more mask locations available, then process 3200 continues atblock 3218. At block 3218, an activation area is calculated. Anactivation area may be an area of the sensor data that has a significantimpact on the output of the neural network structure. Stated somewhatless formally, the activation area may illustrate what areas the neuralnetwork is “using” to classify the sensor data.

The activation area may be calculated at block 3218 using a variety oftechniques. For example, the activation area may be calculated as theset of all pixels stored at any iteration of block 3214. As anotherexample, the activation area may be calculated as the set of all pixelsstored in at least four different iterations of block 3214. In someembodiments, the calculated activation area may be used to generate avisual heatmap for the neural network structure.

In some embodiments, the same neural network structure may have multipleactivation areas. For example, a separate activation area may be definedfor each potential output classification of the neural networkstructure. For example, if the potential classification outputs of theneural network structure are “safe driving,” “texting,” “touchingheadunit,” and “talking,” then there may be four different activationareas for the neural network structure.

In various embodiments, process 3200 may be performed using structuresdescribed elsewhere herein. For example, block 3202 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3204, 3206, 3208, 3210, 3212, 3214, 3216,and 3218 may be performed using computing devices described elsewhereherein (e.g., computing devices 726, 776, 1100, 1316, 1416, 1526, 1626,1726). As a further example, blocks 3204, 3206, 3208, 3210, 3212, 3214,3216, and 3218 may be performed using computing devices describedelsewhere herein (e.g., computing devices 720, 770, 1000, 1320, 1420,1520, 1620, 1720).

FIG. 33 is a diagram of sensor data and activation area calculationaccording to some embodiments of the present disclosure. Image 3302 isan example of sensor data captured of a resource-constrained environment(e.g., as captured at block 2716, block 2808, block 2908). Image 3302shows a driver touching a headunit with his right hand. Image 3304 isthe same as image 3302 except that a mask 3301 is placed over thetop-leftmost corner of the image 3304. While the mask 3301 isillustrated with hashed lines for clarity, the mask 3301 may be allblack pixels in some embodiments. Image 3306 is the same as image 3302and image 3304 except that the mask 3301 is located to the right of thelocation of the mask 3301 on image 3304. Image 3308 is the same is animage 3302, image 3304, and image 3306 except that the mask 3301 islocated to the right of the location of the mask 3301 on image 3304 andimage 3306. Images 3302, 3304, 3306, and 3308 thereby illustrate atechnique for iteratively moving a mask over different locations of asensor data image. In some embodiments, each of image 3302, 3304, 3306,and 3308 may be input to the same neural network structure in order tocalculate the activation area for the “touching headunit” classificationon the neural network structure.

FIG. 34 is a diagram of sensor data and activation area calculationaccording to some embodiments of the present disclosure. Image 3402 isan example of sensor data captured of a resource-constrained environment(e.g., as captured at block 2716, block 2808, block 2908). Image 3402shows a driver touching a headunit with his right hand. Image 3406 is anexample of sensor data captured of a resource-constrained environment(e.g., as captured at block 2716, block 2808, block 2908). Image 3406shows a driver eating with his right and left hands.

Image 3404 and image 3408 are examples of activation areas (e.g., ascalculated at block 3218). Image 3404 is an activation area calculatedbased on the sensor data image 3402. Image 3408 is calculated based onthe sensor data image 3406. In the example activation areas 3404 and3408, each pixel of the activation area is shown with a color intensitybased on the significance of difference contributed to the neuralnetwork structure output data from that pixel. For example, a pixel thatcaused a change in class prediction (e.g., from “touching headunit” to“texting”) may be shown in white. A pixel that did not cause a change inclass prediction but caused a change in confidence level by 25% or moreis shown in light gray. A pixel that did not cause a change in classprediction but caused a change in confidence level by more than 10% butless than 25% is shown in dark gray. And a pixel that did not contributeto a change in class prediction or a change in confidence level of morethan 10% is shown in black.

FIG. 35 is a diagram of sensor data and activation-area expectationaccording to some embodiments of the present disclosure. A calculatedactivation area may be beneficial for a number of reasons in variousembodiments of the present disclosure.

In some embodiments, a calculated activation area may be used todetermine the effectiveness of a neural network structure. In suchembodiments, an expected activation area may be defined. The expectedactivation area may be an area of an image that is expected (e.g., by anexpert operator) to be part of the calculated activation area. Thecalculated activation area may then be calculated after the neuralnetwork structure has been trained. The similarity or difference betweenthe expected activation area and the calculated activation area mayindicate the effectiveness or ineffectiveness of the neural networkstructure.

For example, an expert operator may define an activation area 3501 forthe “touching headunit” classification of a neural network structure asshown overlaid on an image 3502. The expert operator may define theactivation area 3501 because it bounds the headunit area of theautomobile as shown in the image. An expert operator may define anactivate area 3503 for the “eating” classification of a neural networkstructure as shown overlaid on an image 3504. The expert operator maydefine the activate area 3503 because it bounds the area where a driverwould typically have his hands when eating while driving. In someembodiments, an expected activation area may be automaticallycalculated.

Using expected activation areas 3501 and 3503, a neural networkstructure may be evaluated. In particular, a neural network structure isused to create the calculated activation areas 3404 and 3408. Bycomparing the calculated activation areas 3404 and 3408 with theexpected activation areas 3501 and 3503, the neural network structuremay be evaluation.

For example, a comparison of the calculated activation area 3404 for the“touching headunit” classification with the expected activation area3501 for the “touching headunit” classification shows that highest pixelvalues in the calculated activation area 3404 (representing greatersignificance in the activation area) fall within the expected activationarea 3501. This indicates that the neural network structure iseffectively identifying when a driver is touching the headunit.

As another example, a comparison of the calculated activation area 3408for the “eating” classification with the expected activation area 3503for the “eating” classification shows that highest pixel values in thecalculated activation area 3408 (representing greater significance inthe activation area) fall outside the expected activation area 3501. Inparticular, the neural network structure is placing greater significanceon the area down-to-the-left of the driver when predicting the “eating”classification. This is unexpected and indicates that the neural networkstructure is not effectively identifying when a driver is eating.

The result of this comparison may be used in numerous ways. For example,in the case of ineffective identification of a classification, thesystem employing the neural network structure may perform new iterationsof sensor data capture when a driver is eating and retrain the neuralnetwork structure using the newly captured sensor data. This activitymay be performed in order to train the neural network structure to moreeffectively identify the “eating” classification. As another example,the system employing the neural network structure may periodically(e.g., every week) calculate the calculated activation areas and performthe comparison with the expected activation areas in order to monitorthe effectiveness of the neural network structure over time.

In some embodiments, the determination of an ineffective identificationof a classification may result in a notification being provided to theexpert operator. The expert operator may then manually compare thecalculated activation zones to the expected activation zones todetermine where the divergence between the two are occurring. By doingso, the expert operator may identify a new and correct pattern that theneural network structure has detected. As a result, the expert operatormay modify the expected activation zones to account for the newlyidentified pattern.

In some embodiments, the comparison of the calculated activation zone tothe expected activation zone may be performed automatically. Forexample, an activation zone score may be calculated as the percentage ofpixels from the calculated activation zone that fall within the boundsof the expected activation zone. An activation zone score above apredefined threshold (e.g., 50%) may be considered “effective” or“acceptable”, while an activation zone score below a predefinedthreshold (e.g., below 50%) may be considered “ineffective” or“unacceptable.”

In some embodiments, a calculated activation area may be used to reducethe resource demands of the system employing neural network structure.For example, the system may calculate the calculated activation zonesfor all output classifications as described elsewhere herein. The systemmay then determine any pixels that are not in the calculated activationzones for any output classification, which may be referred to in thiscontext as “unused” pixels. Thereafter, when captured sensor data isapplied as input to the neural network structure (e.g., as performed atblocks 2718, 2810), all unused pixels in the captured sensor data may beignored. In this way, the input data to the neural network structure maybe reduced, which may thereby reduce the processing resources andprocessing time needed to apply the captured sensor data as input to theneural network structure. In some instances, the unused pixels mayaccount for a significant portion of the captured sensor data. In someembodiments, unused pixels may also be ignored in subsequent retrainingof the neural network structure.

FIG. 36A is a flowchart of a process 3600 for segmenting sensor data foruse with a neural network in a resource-constrained environmentaccording to some embodiments of the present disclosure. For example,the resource-constrained environment may be an automobile as describedelsewhere in the present disclosure (e.g., automobiles 110, 300, 400,710, 1510, and/or 1610). In some embodiments, the process 3600 may beperformed after having captured sensor data in a resource constrainedenvironment (e.g., block 2702, 2802, 2902) and after having trained aneural network structure based on that sensor data (e.g., block 2710,2804, 2904).

At block 3602, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 3604, the sensor data captured at block 3602 is segmented.Segmenting the sensor data may include deleting a portion of the actualsensor data and replacing it with default value sensor data. Forexample, if the sensor data is an image of the interior of anautomobile, and the image shows both the driver of the automobile and apassenger of the automobile, segmenting the sensor data may includereplacing the captured pixel values in the portion of the image showingthe passenger with a default pixel value (e.g., a black pixel value).

At block 3606, the sensor data segmented at block 3604 is applied to theneural network structure in the resource-constrained environment. Insome embodiments, the process 3600 may include performing other actions,such as described elsewhere herein (e.g., blocks 2020, 2122, 2124, 2220,2222, 2320, 2322, 2420, 2422, 2822, 2824, 2916, 2918).

In various embodiments, process 3600 may be performed using structuresdescribed elsewhere herein. For example, block 3602 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3604 and 3606 may be performed usingcomputing devices described elsewhere herein (e.g., computing devices726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As a further example,blocks 3604 and 3606 may be performed using computing devices describedelsewhere herein (e.g., computing devices 720, 770, 1000, 1320, 1420,1520, 1620, 1720).

FIG. 36B is a flowchart of a process 3650 for segmenting sensor data foruse with a neural network in a resource-constrained environmentaccording to some embodiments of the present disclosure. For example,the resource-constrained environment may be an automobile as describedelsewhere in the present disclosure (e.g., automobiles 110, 300, 400,710, 1510, and/or 1610).

At block 3652, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716).

At block 3654, the sensor data captured at block 3652 is segmented.Segmenting the sensor data may include deleting a portion of the actualsensor data and replacing it with default value sensor data. Forexample, if the sensor data is an image of the interior of anautomobile, and the image shows both the driver of the automobile and apassenger of the automobile, segmenting the sensor data may includereplacing the captured pixel values in the portion of the image showingthe passenger with a default pixel value (e.g., a black pixel value).

At block 3656, a neural network structure is trained in thenon-resource-constrained using the sensor data segmented at block 3654.The training of the neural network structure may be performed asdescribed elsewhere herein (e.g., as described for neural networkstructures 600, 650). In some embodiments, the sensor data captured fromthe resource-constrained environment may be labelled prior to or as partof block 3656.

In some embodiments, the process 3650 may include performing otheractions, such as described elsewhere herein (e.g., blocks 2020, 2122,2124, 2220, 2222, 2320, 2322, 2420, 2422, 2822, 2824, 2916, 2918, 3602,3604, 3606).

In various embodiments, process 3650 may be performed using structuresdescribed elsewhere herein. For example, blocks 3652 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3654 and 3656 may be performed usingcomputing devices described elsewhere herein (e.g., computing devices726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As a further example,blocks 3654 and 3656 may be performed using computing devices describedelsewhere herein (e.g., computing devices 720, 770, 1000, 1320, 1420,1520, 1620, 1720).

In some embodiments, the processes 3600 and/or 3650 may be used as partof or in conjunction with other processes described elsewhere herein(e.g., processes 2000, 2100, 2200, 2300, 2400, 2800, 2900, 3200, 3600,3650). For example, in some embodiments, process 3650 may be performedprior to performing process 3600.

FIG. 37 is a flowchart of a process 3700 for segmenting sensor data foruse with a neural network in a resource-constrained environmentaccording to some embodiments of the present disclosure. For example,the resource-constrained environment may be an automobile as describedelsewhere in the present disclosure (e.g., automobiles 110, 300, 400,710, 1510, and/or 1610). In some embodiments, the process 3700 may beperformed as part of a segmenting activity described elsewhere herein(e.g., blocks 3604, 3654).

At block 3702, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). In some embodiments the sensor data may be an imageof an interior of an automobile. In some embodiments, the image of theinterior of the automobile may include a driver present in theautomobile.

At block 3704, skin tones are identified in the sensor data. Block 3704may include using an image processor to identify locations in the imagewhere colors matching typical human skin tones are present. In someembodiments, a face of a driver of the automobile may be determinedbased on the identification of skin tones in the image. For example,block 3704 may include identifying all clusters of pixels with a humanskin tone consisting of at least a predefined minimum number of pixels(e.g., at least 20 pixels). Continuing the example, block 3704 mayinclude identifying one of the clusters as being the face of the driverbased on the locations of the clusters (e.g., selecting the largestcluster in the top-right quadrant of the image as the face of thedriver).

At block 3706, eyes are located in the sensor data. Block 3706 mayinclude using a facial recognition algorithm to identify the location ofa face and the associated eyes within the image. In some embodiments,the facial recognition algorithm performed at block 3706 may beperformed in only portions of the image. For example, the facialrecognition algorithm may only be used to locate human eyes in orimmediately adjacent to clusters of pixels identified at block 3704 ascontaining human skin tones. As another example, the facial recognitionalgorithm may only be used to locate human eyes in or immediatelyadjacent to a cluster of pixels identified at block 3704 ascorresponding to the face of the driver of the automobile. In at leastthat way, the performance of block 3704 may be used to reduce the searchspace for the location of eyes at block 3706.

At block 3708, shoulders are located in the sensor data. Block 3708 mayinclude estimating the location of the shoulders based on the locationof the eyes determined at block 3706. For example, block 3708 mayinclude using an image processor to identify concave shapes facingoutward in a space immediately below and to the sides of the face and/oreyes identified at blocks 3704 and 3706, respectively. As anotherexample, block 3708 may include measuring down and outward from each eyeidentified at block 3706 for a predefined number of pixels (e.g., 30pixels), and marking the resulting locations as the shoulders. In someembodiments, the technique of measuring a distance from the eyes toidentify the shoulders may include using a dynamic number of pixels forthe measurement (e.g., based on the image size, based on the number ofpixels between the eyes identified at block 3706, and/or based on thenumber of pixels contained in the face cluster identified at block3704).

At block 3710, sensor data content is removed other than a person. Block3710 may include masking the image of the interior of the automobile tothe driver. For example, masking the image to the driver may includereplacing all pixel values outside the area of the image occupied by thedriver with a default pixel value (e.g., black pixel value). In someembodiments, the block 3710 may be performed based on the identificationof skin tones, face, eyes, and/or shoulders as performed at blocks 3704,3704, 3706, 3708, respectively. For example, block 3710 may includemeasuring a predefined number of pixels out from the locations of thelocations of the eyes and shoulders as identified at block 3706 and3708, respectively. As another example, block 3710 may include providingthe image as well as the location of the eyes and/or the location of theshoulders as input to an image processor, the image processor whichidentify the outline of a human based on the provided input. In someembodiments, the image may be reduced in size by cropping the image tosmaller dimensions to remove the content other than the person.

In various embodiments, process 3700 may allow a neural networkstructure to be more accurate. In particular, as described elsewhereherein (e.g., processes 1800, 1900, 2000, 2100, 2200, 2300, 2400, 2700,2800, 2900, 3200, 3600, 3650), sensor data may be used to train a neuralnetwork structure and/or provided as input to an already trained neuralnetwork structure. When the sensor data includes extraneous aspects notpertinent to the accurate determination of a driver or automobile state,the extraneous aspects of the sensor data may cause the neural networkstructure to be less accurate. For instance, in a case where the neuralnetwork structure is used to determine if the driver is in a “safe” orin an “unsafe” state, but the images (i.e., the sensor data) are colorimages of the entire interior of the automobile, the neural networkstructure may be less accurate. For example, as a result of training,the neural network structure may inadvertently use the upholstery coloror the position of the passenger headrest as a factor in determining thestate of the driver. As another example, if the neural network structureis trained using images from a small compact car, the neural networkstructure may be inaccurate in predicting driver state when the imagesapplied as input are captured in a large pickup truck. By removingsensor data content other than the driver, the neural network structuremay be more accurately trained and more accurately applied to determinethe driver state only based on relevant aspects of the driver's body.Similar benefits inure when segmenting an image to include only relevantaspects of the interior of an automobile when the neural network istrained and applied to detect deviations from normal configurations ofthe interior of the automobile.

In various embodiments, process 3700 may be performed using structuresdescribed elsewhere herein. For example, block 3702 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3704, 3706, 3708, and 3710 may be performedusing computing devices described elsewhere herein (e.g., computingdevices 726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As a furtherexample, blocks 3704, 3706, 3708, 3710 may be performed using computingdevices described elsewhere herein (e.g., computing devices 720, 770,1000, 1320, 1420, 1520, 1620, 1720).

FIG. 38 is a diagram of sensor data and sensor data segmenting accordingto some embodiments of the present disclosure. In some embodiments, theprocess 3700 may be performed as part of a segmenting activity describedelsewhere herein (e.g., blocks 3604, 3654, process 3700).

Image 3802 is an image of an interior of an automobile. Image 3802includes a driver (man to the right) and a passenger (woman to theleft). Image 3802 may be captured by any of a variety of sensors, suchas described with respect to sensors elsewhere in the present disclosure(e.g., sensors 232, 234, 460, 472, 716, 800, 1516, 1518, 1616, 1716). Insome embodiments, image 3802 may be captured as sensor data at block3702 as part of process 3700.

Image 3804 is an image of an interior of an automobile that may be usedfor identifying human skin tones. Image 3804 may be generated as atransformation of image 3802. For example, image 3804 may be generatedfrom image 3802 by increasing the amplitude, contrast, or othercharacteristic of pixels with a predefined pixel value corresponding toa human skin tone. In some embodiments, image 3804 may be used toidentify skin tones and/or a face at block 3704 as part of process 3700.

Image 3806 is a model of the interior of an automobile that estimatesthe location of eyes and shoulders of a person. Image 3806 includes anestimation of a person's eyes (i.e., line 3822, the short horizontalline towards top-center of image 3806). Image 3806 includes anestimation of a person's shoulders (i.e., line 3826, the long horizontalline towards middle of image 3806). Image 3806 includes an estimation ofa person's torso (i.e., line 3824, the long vertical line in middle ofimage 3806). In some embodiments, image 3806 may be generated using animage processor so as to identify the location of a person in image3802. In some embodiments, image 3806 may be generated at blocks 3706and 3708 as part of process 3700.

Image 3808 is an image of the interior of an automobile that has beensegmented to the driver. Image 3808 includes the portions of image 3802that correspond to the driver of the automobile. However, image 3808 hasbeen masked for portions of image 3802 that do not correspond to thedriver of the automobile. In image 3808, the non-driver portions of theimage are masked by changing the pixels in those portions to all beblack. In some embodiments, image 3808 may be generated at block 3710 aspart of process 3700.

In embodiments of the present disclosure, the segmenting of sensor datadescribed herein (e.g., processes 3600, 3650, 3700) may provide variousbenefits. First, the segmenting of sensor data may reduce the size ofthe sensor data, and thereby reduce the computation time and/orresources needed to train a neural network structure. Second, thesegmenting of sensor data may reduce the size of the sensor data, andthereby reduce the computation time and/or resources needed to apply thesensor data as input to a neural network structure. In such cases, theproviding of segmented sensor data as input to the neural networkstructure may allow more effective use of the limited computingresources available in the resource-constrained environment.Additionally, the providing of segmented sensor data as input to theneural network structure may allow faster determination of a result ofthe application of the sensor data as input to the neural networkstructure, and thus action may be taken more rapidly based on the resultof the application of the sensor data as input to the neural networkstructure. This may be especially beneficial in someresource-constrained environments (e.g., an automobile), where theactions taken based on the results of the application of the sensor dataas input to the neural network structure may include actions that impactthe safety of a person (e.g., reducing the velocity of an automobile).

Third, the segmenting of sensor data may improve the effectiveness ofthe neural network structure. For example, if the neural networkstructure is used to determine the state of the driver of an automobile,it is possible that the neural network structure may provide someweighting (i.e., give some “importance”) to the values of the pixels inthe area of the front passenger seat. However, the portion of the imagecovering the front passenger seat may have little predictive value as tothe state of the driver of the automobile. Thus, any weighting appliedto the pixels in that portion of the image may create inaccuracies inthe results of the neural network structure in determining the state ofthe driver of the automobile. In such a case, the segmenting of theimage to remove the captured pixel values from the portion of the imagecovering the front passenger seat may remove this inaccurate weightingand thereby improve the effectiveness of the neural network structure indetermining the state of the driver of the automobile.

FIG. 39A is a flowchart of a process 3900 for training a neural networkin a resource-constrained environment using keypoints according to someembodiments of the present disclosure. For example, theresource-constrained environment may be an automobile as describedelsewhere in the present disclosure (e.g., automobiles 110, 300, 400,710, 1510, and/or 1610).

At block 3902, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). For example, the sensor data may be an image of theinterior of an automobile.

At block 3904, keypoints are defined for the sensor data. A keypoint maybe a location in sensor data. For example, a keypoint may be defined byan x coordinate and a y coordinate in an area occupied by sensor data(e.g., a rectangular image). One or more pixels of sensor data maycorrespond to a keypoint. For example, a pixel located at the same x, ycoordinates as the keypoint may be a pixel corresponding to thekeypoint. Block 3904 may include defining one keypoint corresponding tothe sensor data or defining more than one keypoint corresponding to thesensor data. In some embodiments, block 3904 may include a human userdefining a keypoint. In some embodiments, block 3904 may includeautomatically defining a keypoint (e.g., based on a calculatedactivation area, and/or based on an expected activation area).

At block 3906, a neural network structure is trained in thenon-resource-constrained using the sensor data captured at block 3902and using the keypoints defined at block 3904. The training of theneural network structure may be performed as described elsewhere herein(e.g., as described for neural network structures 600, 650). In someembodiments, the sensor data captured from the resource-constrainedenvironment may be labelled prior to or as part of block 3906.

In some embodiments, block 3906 may include modifying the training ofthe neural network structure. For example, in some embodiments, a pixelcorresponding to a keypoint may have its value increased by a fixedmultiple (e.g., two times its original value) prior to using the pixelas input to the neural network training algorithm. As another example,in some embodiments, a pixel corresponding to a keypoint may beduplicated (e.g., three additional times) prior to using the pixel asinput to the neural network training algorithm. Block 3906 may includealtering the input to a neural network structure training algorithmbased on the keypoints defined at block 3904 in various other waysconsistent with the present disclosure.

In some embodiments, block 3906 may include modifying a neural networkstructure. For example, in some embodiments, block 3906 may includetraining a neural network structure using the sensor data captured atblock 3902, and then modifying a weight value for a connection of theneural network structure over which a pixel corresponding to a keypointpasses (e.g., doubling the weight on a connection carrying the pixelcorresponding to a keypoint). As another example, in some embodiments,block 3906 may include training a neural network structure using thesensor data captured at block 3902, and then modifying an activationfunction for a node of the neural network structure into which a pixelcorresponding to a keypoint passes (e.g., increasing the likelihood ofactivation for the connection carrying the pixel corresponding to akeypoint). Block 3906 may include altering a trained neural networkstructure based on the keypoints defined at block 3904 in various otherways consistent with the present disclosure.

In some embodiments, the process 3900 may include performing otheractions, such as described elsewhere herein (e.g., blocks 2020, 2122,2124, 2220, 2222, 2320, 2322, 2420, 2422, 2822, 2824, 2916, 2918, 3602,3604, 3606).

In various embodiments, process 3900 may be performed using structuresdescribed elsewhere herein. For example, block 3902 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3904 and 3906 may be performed usingcomputing devices described elsewhere herein (e.g., computing devices726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As a further example,blocks 3904 and 3906 may be performed using computing devices describedelsewhere herein (e.g., computing devices 720, 770, 1000, 1320, 1420,1520, 1620, 1720).

FIG. 39B is a flowchart of a process 3950 for training a neural networkin a resource-constrained environment using keyzones according to someembodiments of the present disclosure. For example, theresource-constrained environment may be an automobile as describedelsewhere in the present disclosure (e.g., automobiles 110, 300, 400,710, 1510, and/or 1610).

At block 3952, sensor data is captured in the resource-constrainedenvironment. The sensor data may be captured by any of a variety ofsensors, such as described with respect to sensors elsewhere in thepresent disclosure (e.g., sensors 232, 234, 460, 472, 716, 800, 1516,1518, 1616, 1716). For example, the sensor data may be an image of theinterior of an automobile.

At block 3954, keyzones are defined for the sensor data. A keyzone maybe an area in sensor data. For example, a keyzone may be defined by fourpairs of x, y coordinates (e.g., a rectangular bounding box) in an areaoccupied by sensor data (e.g., a rectangular image). One or more pixelsof sensor data may correspond to a keyzone. For example, a group of 16pixels located within the area bounded by the keyzone may be pixelscorresponding to the keyzone. Block 3954 may include defining onekeyzone corresponding to the sensor data or defining more than onekeyzone corresponding to the sensor data. In some embodiments, block3954 may include a human user defining a keyzone. In some embodiments,block 3954 may include automatically defining a keyzone (e.g., based ona calculated activation area, and/or based on an expected activationarea).

At block 3956, a neural network structure is trained in thenon-resource-constrained using the sensor data captured at block 3952and using the keyzones defined at block 3954. The training of the neuralnetwork structure may be performed as described elsewhere herein (e.g.,as described for neural network structures 600, 650). In someembodiments, the sensor data captured from the resource-constrainedenvironment may be labelled prior to or as part of block 3956.

In some embodiments, block 3956 may include modifying the training ofthe neural network structure. For example, in some embodiments, pixelscorresponding to a keyzone may have their values increased by a fixedmultiple (e.g., two times its original value) prior to using the pixelsas input to the neural network training algorithm. As another example,in some embodiments, pixels corresponding to a keyzone may be duplicated(e.g., three additional times) prior to using the pixels as input to theneural network training algorithm. Block 3956 may include altering theinput to a neural network structure training algorithm based on thekeyzones defined at block 3954 in various other ways consistent with thepresent disclosure.

In some embodiments, block 3956 may include modifying a neural networkstructure. For example, in some embodiments, block 3956 may includetraining a neural network structure using the sensor data captured atblock 3952, and then modifying a weight value for a connection of theneural network structure over which a pixel corresponding to a keyzonepasses (e.g., doubling the weight on a connection carrying a pixelcorresponding to a keyzone). As another example, in some embodiments,block 3956 may include training a neural network structure using thesensor data captured at block 3952, and then modifying an activationfunction for a node of the neural network structure into which a pixelcorresponding to a keyzone passes (e.g., increasing the likelihood ofactivation for the connection carrying a pixel corresponding to akeyzone). Block 3956 may include altering a trained neural networkstructure based on the keyzones defined at block 3954 in various otherways consistent with the present disclosure.

In some embodiments, the process 3950 may include performing otheractions, such as described elsewhere herein (e.g., blocks 2020, 2122,2124, 2220, 2222, 2320, 2322, 2420, 2422, 2822, 2824, 2916, 2918, 3602,3604, 3606).

In various embodiments, process 3950 may be performed using structuresdescribed elsewhere herein. For example, block 3952 may be performedusing sensors described elsewhere herein (e.g., sensors 460, 472, 716,765, 766, 767, 768, 769, 800, 1316, 1416, 1516, 1518, 1616, 1618, 1716).As a further example, blocks 3954 and 3956 may be performed usingcomputing devices described elsewhere herein (e.g., computing devices726, 776, 1100, 1316, 1416, 1526, 1626, 1726). As a further example,blocks 3954 and 3956 may be performed using computing devices describedelsewhere herein (e.g., computing devices 720, 770, 1000, 1320, 1420,1520, 1620, 1720).

In some embodiments, the processes 3900 and/or 3950 may be used as partof or in conjunction with other processes described elsewhere herein(e.g., processes 2000, 2100, 2200, 2300, 2400, 2800, 2900, 3200, 3600,3650, 3900, 3950). For example, in some embodiments, processes 3900 and3950 may be combined so that both keypoints and keyzones are defined,and so that a neural network is trained based on captured sensor data,keypoints, and keyzones.

FIG. 40 is a diagram of an annotation interface 4000 according to someembodiments of the present disclosure. In some embodiments, theannotation interface 4000 may be used to annotate sensor data withkeypoints. The annotation interface 4000 may allow a user to annotate animage with one or more keypoints (e.g., six keypoints as illustrated).In some embodiments, the annotation interface 4000 may be configured forthe user to define a keypoint by clicking a point on the image. In suchembodiments, the x, y coordinates of the location of the user's clickmay be determined and assigned as the location of the keypoint. In suchembodiments, the annotation interface 4000 may output the x, ycoordinates of the keypoint for view by the user after assigning the x,y coordinates to the keypoint. In some embodiments, the annotationinterface 4000 may be configured for the user to define a keypoint byentering an x coordinate value and entering a y coordinate value. Insome embodiments, the annotation interface 4000 may be used to definekeypoints as part of other processes described herein (e.g., block3904).

FIG. 41 is a diagram of an annotation interface 4100 according to someembodiments of the present disclosure. In some embodiments, theannotation interface 4100 may be used to annotate sensor data withkeyzones. The annotation interface 4100 may allow a user to annotate animage with one or more keyzones (e.g., six keyzones as illustrated). Insome embodiments, the annotation interface 4100 may allow the user todefine a keyzone as a geometric shape (e.g., a bounding box asillustrated). In some embodiments, the annotation interface 4100 may beconfigured for the user to define a keyzone by clicking a point on theimage and dragging the pointer to a second location on the image. Insome embodiments, the annotation interface 4100 may be configured forthe user to define a keyzone by clicking three or more points on theimage. In some embodiments, the x, y coordinates of the keyzone may bedetermined based on the user's clicks and or dragging, and the x, ycoordinates may be assigned as the vertices of the keyzone. In suchembodiments, the annotation interface 4100 may output the x, ycoordinates of the vertices of the keyzone for view by the user afterassigning the x, y coordinates to the keyzone. In some embodiments, theannotation interface 4100 may be configured for the user to define akeyzone by entering an x coordinate values and entering a y coordinatevalues for the vertices of the keyzone. In some embodiments, theannotation interface 4100 may be used to define keyzones as part ofother processes described herein (e.g., block 3954).

In some embodiments of the present disclosure, the keypoints and/orkeyzones may be beneficial for numerous reasons. The keypoints and/orkeyzones may be effective to improve the effectiveness of a neuralnetwork structure. For example, by using the keypoints and/or keyzonesto train a neural network structure or modify a trained neural networkstructure, such as described elsewhere herein (e.g., blocks 3906, 3956),the neural network structure may be more effective in determining aresult based on the input (e.g., determining a driver state based on aninput image of the interior of the automobile). This benefit may arise,at least in part, because the keypoints and/or keyzones may define areaof particular importance to determining the neural networks output(e.g., the driver's face and the driver's hands for determining whetherthe driver state is “safe driving,” “texting,” “touching headunit,” or“talking”). The keypoints and/or keyzones may therefore incorporateadditional knowledge about the environment on which the neural networkstructure is operating that may be incorporated into the neural networkstructure. In cases where the keypoints and/or keyzones are defined by ahuman user, the keypoints and/or keyzones may incorporate the humanuser's knowledge about the environment (e.g., the human user's knowledgethat the face and hands are important to determining the driver state).In cases where the keypoints and/or keyzones are defined automatically,the keypoints and/or keyzones may incorporate knowledge determined bysome computational process (e.g., using the calculated activation zoneto incorporate knowledge learned by the system from the previousapplication of the neural network structure).

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

We claim:
 1. A system comprising: a sensor located in aresource-constrained environment configured to generate first sensordata of the resource-constrained environment and second sensor data ofthe resource-constrained environment; a first computing device notlocated in the resource-constrained environment configured to produce aneural network structure based on the first sensor data; a secondcomputing device located in the resource-constrained environmentconfigured to provide the second sensor data as input to the neuralnetwork structure, wherein the second computing device is furtherconfigured to determine a state of the resource-constrained environmentbased on the input of the second sensor data to the neural networkstructure; and a controller located in the resource-constrainedenvironment configured to control a device in the resource-constrainedenvironment based on the state of the resource-constrained environmentdetermined by the second computing device, wherein the second computingdevice is further configured to calculate an activation area for theneural network structure, wherein the second computing device inconfigured to calculate the activation area for the neural networkstructure, at least in part, by providing third sensor data as input tothe neural network structure, wherein the third sensor data is formed byplacing a first mask at a first location in the second sensor data,wherein the second computing device in configured to calculate theactivation area for the neural network structure, at least in part, byproviding fourth sensor data as input to the neural network structure,wherein the fourth sensor data is formed by placing a second mask at asecond location in the second sensor data, wherein the second computingdevice in configured to calculate the activation area for the neuralnetwork structure, at least in part, by providing fifth sensor data asinput to the neural network structure, wherein the fifth sensor data isformed by placing a third mask at a third location in the second sensordata, wherein the second computing device in configured to calculate theactivation area for the neural network structure, at least in part, bycomparing a first result of providing the second sensor data as input tothe neural network structure to a second result of providing the thirdsensor data as input to the neural network structure, wherein the secondcomputing device in configured to calculate the activation area for theneural network structure, at least in part, by comparing the firstresult of providing the second sensor data as input to the neuralnetwork structure to a third result of providing the fourth sensor dataas input to the neural network structure, and wherein the secondcomputing device in configured to calculate the activation area for theneural network structure, at least in part, by comparing the firstresult of providing the second sensor data as input to the neuralnetwork structure to a fourth result of providing the fifth sensor dataas input to the neural network structure.
 2. The system of claim 1,wherein the second computing device is configured to calculate theactivation area for a first prediction class of the neural networkstructure, and wherein the second computing device is configured tocalculate a second activation area for a second prediction class of theneural network structure.
 3. The system of claim 2, wherein the secondcomputing device is configured to compare the activation area to anexpected activation area for the first prediction class; and wherein thesecond computing device is configured to compare the second activationarea to an expected activation area for the second prediction class. 4.The system of claim 3, wherein the second computing device isconfigured, based at least in part on a result of comparing theactivation area to the expected activation area for the first predictionclass, to instruct the sensor to generate additional sensor data for thefirst prediction class.
 5. The system of claim 1, wherein the secondcomputing device is configured to compare the first result to the secondresult, at least in part, by determining if a prediction class providedas part of the second result is different from a prediction classprovided as part of the first result.
 6. The system of claim 1, whereinthe second computing device is configured to compare the first result tothe second result, at least in part, by determining if a secondprediction confidence level provided as part of the second result isdifferent from a first prediction confidence level provided as part ofthe first result.
 7. The system of claim 6, wherein the second computingdevice is configured to determine if the second prediction confidencelevel is different from the first prediction confidence level, at leastin part, by determining if the difference between the second predictionconfidence level and the first prediction confidence level is greaterthan a predefined threshold value.
 8. The system of claim 1, wherein thesecond computing device is configured to determine an effectivenessvalue for the neural network structure based at least in part on thecalculated activation area.
 9. The system of claim 8, wherein the secondcomputing device is configured, based at least in part on the determinedeffectiveness value, to instruct the first computing device to produce asecond neural network structure.
 10. A method comprising: generatingfirst sensor data and second sensor data of a resource-constrainedenvironment using a sensor located in the resource-constrainedenvironment; producing a neural network structure based on the firstsensor data using a first computing device not located in theresource-constrained environment; inputting the sensor data to theneural network structure using a second computing device located in theresource-constrained environment; determining a state of theresource-constrained environment using the second computing device basedon the input of the second sensor data to the neural network structure;controlling an indicator device in the resource-constrained environmentusing a controller located in the resource-constrained environment basedon the state of the resource-constrained environment determined by thesecond computing device; and calculating an activation area for theneural network structure; wherein calculating the activation area forthe neural network structure comprises providing third sensor data asinput to the neural network structure, wherein the third sensor data isformed by placing a first mask at a first location in the second sensordata, wherein calculating the activation area for the neural networkstructure comprises providing fourth sensor data as input to the neuralnetwork structure, wherein the fourth sensor data is formed by placing asecond mask at a second location in the second sensor data, whereincalculating the activation area for the neural network structurecomprises providing fifth sensor data as input to the neural networkstructure, wherein the fifth sensor data is formed by placing a thirdmask at a third location in the second sensor data, wherein calculatingthe activation area for the neural network structure comprises comparinga first result of providing the second sensor data as input to theneural network structure to a second result of providing the thirdsensor data as input to the neural network structure, whereincalculating the activation area for the neural network structurecomprises comparing the first result of providing the second sensor dataas input to the neural network structure to a third result of providingthe fourth sensor data as input to the neural network structure, andwherein calculating the activation area for the neural network structurecomprises comparing the first result of providing the second sensor dataas input to the neural network structure to a fourth result of providingthe fifth sensor data as input to the neural network structure.
 11. Themethod of claim 10, wherein calculating the activation area for theneural network structure comprises calculating the activation area for afirst prediction class of the neural network structure, the methodfurther comprising: calculating a second activation area for a secondprediction class of the neural network structure.
 12. The method ofclaim 11, further comprising: comparing the activation area to anexpected activation area for the first prediction class; and comparingthe second activation area to an expected activation area for the secondprediction class.
 13. The method of claim 12, further comprising:generating, based at least in part on a result of the comparing theactivation area to the expected activation area for the first predictionclass, additional sensor data for the first prediction class.
 14. Themethod of claim 10, wherein comparing the first result to the secondresult comprises determining if a prediction class provided as part ofthe second result is different from a prediction class provided as partof the first result.
 15. The method of claim 10, wherein comparing thefirst result to the second result comprises determining if a secondprediction confidence level provided as part of the second result isdifferent from a first prediction confidence level provided as part ofthe first result.
 16. The method of claim 15, wherein determining if thesecond prediction confidence level is different from the firstprediction confidence level comprises determining if the differencebetween the second prediction confidence level and the first predictionconfidence level is greater than a predefined threshold value.
 17. Themethod of claim 10, further comprising: determining an effectivenessvalue for the neural network structure, based at least in part on thecalculated activation area.
 18. The method of claim 17, furthercomprising: instructing the first computing device to produce a secondneural network structure, based at least in part on the determinedeffectiveness value.